Python抓取JSON转换为CSV [英] Python scraping JSON converting to CSV
本文介绍了Python抓取JSON转换为CSV的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我对python还是很陌生，这可能是一个可怕的问题，但是在这里:
I'm very new to python and this will probably be a horrendous question, but here it goes:
我正在尝试从特定网站抓取数据，并将该数据转换为CSV.我设法找到了一个有效的抓取工具"，并将文件保存在我的计算机上.但是，当我尝试再次将JSON加载到python中时，会遇到各种错误.我相信其中一个错误是由于我在一个文件中添加多个JSON数组而导致的，而json.load()只能加载一个JSON(据我了解).
I'm trying to scrape data from a particular website and have that data converted to CSV. I've managed to get a working "scraper" and save the file on my computer. However, when I try to load the JSON into python again I get all kinds of errors. I believe one of the errors is caused due to the fact that I'm adding multiple JSON arrays in one file and json.load() can only load a single JSON (from what I understood).
长话短说:有人可以指导我正确地抓取数据并获取可读性强的CSV的正确方向吗?
To make a long story short: Could anyone please guide me in the proper direction as to how I scrape this data and get a nice readable CSV?
请在下面找到我的代码.
Please find my code below.
import urllib.request
import urllib.parse
import datetime
import time
import json
import csv
def grabData():
startdate = datetime.date(2015,2,1)
startdate += datetime.timedelta(days=1)
datalist = []
for i in range(2):
unixtime = time.mktime(startdate.timetuple())
print(unixtime)
url = 'https://api.*************/34.700869,135.503755,'+str(int(unixtime))+'?units=si'
print(url)
webURL = urllib.request.urlopen(url)
data = webURL.read()
print(data)
encoding = webURL.info().get_content_charset('utf-8')
json.loads(data.decode(encoding))
datalist.append(data.decode())
startdate += datetime.timedelta(days=1)
text = datalist
saveFile = open('osakatest.json','a')
saveFile.write(str(text))
saveFile.close()
grabData()
这是我得到的输出的摘要.
Here is a snippet of the output I got.
[{"latitude":34.700869,"longitude":135.503755,"timezone":"Asia/Tokyo","offset":9,"currently":{"time":1422831600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":3.51,"apparentTemperature":1.37,"dewPoint":-2.53,"humidity":0.65,"windSpeed":2.24,"windBearing":294,"visibility":11.76,"pressure":1028.66},"hourly":{"summary":"Clear throughout the day.","icon":"clear-day","data":[{"time":1422802800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.94,"apparentTemperature":1.07,"dewPoint":-2.09,"humidity":0.7,"windSpeed":1.91,"windBearing":295,"visibility":14.52,"pressure":1029.11},{"time":1422806400,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.65,"dewPoint":-2.12,"humidity":0.7,"windSpeed":2.23,"windBearing":292,"visibility":14.52,"pressure":1028.7},{"time":1422810000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.78,"dewPoint":-2.17,"humidity":0.69,"windSpeed":2.11,"windBearing":270,"visibility":14.16,"pressure":1028.52},{"time":1422813600,"summary":"Partly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.47,"dewPoint":-2.12,"humidity":0.7,"windSpeed":2.36,"windBearing":282,"visibility":15.77,"cloudCover":0.38,"pressure":1028.25},{"time":1422817200,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.65,"dewPoint":-2.13,"humidity":0.7,"windSpeed":2.19,"windBearing":285,"visibility":14.52,"pressure":1028.16},{"time":1422820800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.56,"dewPoint":-1.72,"humidity":0.72,"windSpeed":2.32,"windBearing":301,"visibility":12.41,"pressure":1028.21},{"time":1422824400,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.55,"dewPoint":-1.82,"humidity":0.71,"windSpeed":2.28,"windBearing":295,"visibility":14.95,"cloudCover":0.75,"pressure":1028.25},{"time":1422828000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.97,"apparentTemperature":0.69,"dewPoint":-2.1,"humidity":0.69,"windSpeed":2.29,"windBearing":304,"visibility":12.41,"pressure":1028.55},{"time":1422831600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":3.51,"apparentTemperature":1.37,"dewPoint":-2.53,"humidity":0.65,"windSpeed":2.24,"windBearing":294,"visibility":11.76,"pressure":1028.66},{"time":1422835200,"summary":"Mostly Cloudy","icon":"partly-cloudy-day","precipType":"rain","temperature":4.28,"apparentTemperature":2.64,"dewPoint":-3.14,"humidity":0.58,"windSpeed":1.89,"windBearing":285,"visibility":14.63,"cloudCover":0.74,"pressure":1028.55},{"time":1422838800,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":4.95,"apparentTemperature":2.61,"dewPoint":-3.67,"humidity":0.54,"windSpeed":2.79,"windBearing":301,"visibility":11.76,"pressure":1028.43},{"time":1422842400,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":6.13,"apparentTemperature":3.48,"dewPoint":-3.35,"humidity":0.51,"windSpeed":3.6,"windBearing":316,"visibility":11.76,"pressure":1027.97},{"time":1422846000,"summary":"Mostly Cloudy","icon":"partly-cloudy-day","precipType":"rain","temperature":5.99,"apparentTemperature":3.08,"dewPoint":-3.91,"humidity":0.49,"windSpeed":3.99,"windBearing":320,"visibility":14.45,"cloudCover":0.75,"pressure":1027.18},{"time":1422849600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":6.01,"apparentTemperature":3.01,"dewPoint":-3.74,"humidity":0.5,"windSpeed":4.17,"windBearing":329,"visibility":11.76,"pressure":1026.79},{"time":1422853200,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.94,"apparentTemperature":3.4,"dewPoint":-3.33,"humidity":0.51,"windSpeed":3.34,"windBearing":343,"visibility":11.76,"pressure":1026.69},{"time":1422856800,"summary":"Overcast","icon":"cloudy","precipType":"rain","temperature":5.78,"apparentTemperature":2.98,"dewPoint":-3.41,"humidity":0.52,"windSpeed":3.72,"windBearing":342,"visibility":14.71,"cloudCover":0.96,"pressure":1026.77},{"time":1422860400,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.74,"apparentTemperature":3.42,"dewPoint":-3.07,"humidity":0.53,"windSpeed":2.97,"windBearing":337,"visibility":11.72,"pressure":1026.87},{"time":1422864000,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.33,"apparentTemperature":2.86,"dewPoint":-2.67,"humidity":0.56,"windSpeed":3.07,"windBearing":329,"visibility":11.76,"pressure":1027.16},{"time":1422867600,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":4.43,"apparentTemperature":1.56,"dewPoint":-1.98,"humidity":0.63,"windSpeed":3.37,"windBearing":323,"visibility":14.45,"cloudCover":0.75,"pressure":1027.26},{"time":1422871200,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.36,"apparentTemperature":1.63,"dewPoint":-1.97,"humidity":0.63,"windSpeed":3.14,"windBearing":315,"visibility":11.76,"pressure":1027.55},{"time":1422874800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.25,"apparentTemperature":1.71,"dewPoint":-1.68,"humidity":0.65,"windSpeed":2.87,"windBearing":300,"visibility":12.41,"pressure":1027.77},{"time":1422878400,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":4.27,"apparentTemperature":1.95,"dewPoint":-1.74,"humidity":0.65,"windSpeed":2.59,"windBearing":313,"visibility":15.8,"cloudCover":0.76,"pressure":1027.98},{"time":1422882000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.25,"apparentTemperature":1.78,"dewPoint":-1.73,"humidity":0.65,"windSpeed":2.77,"windBearing":297,"visibility":14.52,"pressure":1028.02},{"time":1422885600,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.24,"apparentTemperature":2.03,"dewPoint":-1.72,"humidity":0.65,"windSpeed":2.46,"windBearing":296,"visibility":14.52,"pressure":1028.12}]},"daily":{"data":[{"time":1422802800,"summary":"Clear throughout the day.","icon":"clear-day","sunriseTime":1422827846,"sunsetTime":1422865697,"moonPhase":0.44,"precipType":"rain","temperatureMin":2.84,"temperatureMinTime":1422824400,"temperatureMax":6.13,"temperatureMaxTime":1422842400,"apparentTemperatureMin":0.47,"apparentTemperatureMinTime":1422813600,"apparentTemperatureMax":3.48,"apparentTemperatureMaxTime":1422842400,"dewPoint":-2.49,"humidity":0.62,"windSpeed":2.62,"windBearing":310,"visibility":13.45,"cloudCover":0.73,"pressure":1027.9}]},"flags":{"sources":["isd"],"isd-stations":["476490-99999","476510-99999","476630-99999","476840-99999","477500-99999","477590-99999","477610-99999","477690-99999","477700-99999","477710-99999","477720-99999","477740-99999","477760-99999","477770-99999","477790-99999","477800-99999"],"units":"si"}}]
理想的CSV输出:
time,summary,icon,precipType,temperature,apparentTemperature,dewPoint,humidity,windSpeed,windBearing,visibility,pressure,cloudCover
1422802800,Clear,clear-night,rain,2.94,1.07,-2.09,0.7,1.91,295,14.52,1029.11,
1422806400,Clear,clear-night,rain,2.88,0.65,-2.12,0.7,2.23,292,14.52,1028.7,
1422810000,Clear,clear-night,rain,2.88,0.78,-2.17,0.69,2.11,270,14.16,1028.52,
1422813600,Partly Cloudy,partly-cloudy-night,rain,2.84,0.47,-2.12,0.7,2.36,282,15.77,1028.25,0.38
推荐答案
我假设您已经完成了网站的解析工作，并且您拥有包含python dict的最终list.我将仅在解决方案中使用最终输出.
I'm assuming you did the part of parsing the website and you have the final list which contain a python dict. I'll use only the final output in my solution.
因此，您可以尝试使用这些代码.他们将解析您当前的list并返回csv.
So, you can try those codes. They will parse your current list and return a csv.
PS:
像您一样处理dict并用我当前的代码解析它可能不会按照您在问题中给出的顺序输出输出.
Handling a dict like yours and parsing it with my current codes may not output an output with the order you gave in your question.
否则，您可以使用和修改这些代码，直到它们满足您的需求为止.
Otherwise, you can use and modify those codes untill they fills your needs.
但是，使用当前的解决方案，您可以拥有正确的csv.如果您在以下代码中发现了任何错误或无法理解的错误，请随时发表评论.
however, with this current solutions you can have a correct csv. If you found any bugs or something you can't understand in the code below, don't hesitate to leave your comment.
解决方案1:使用pandas:
Solution1: Using pandas:
import pandas as pd
from os import getcwd
a = [{"latitude":34.700869,"longitude":135.503755,"timezone":"Asia/Tokyo","offset":9,"currently":{"time":1422831600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":3.51,"apparentTemperature":1.37,"dewPoint":-2.53,"humidity":0.65,"windSpeed":2.24,"windBearing":294,"visibility":11.76,"pressure":1028.66},"hourly":{"summary":"Clear throughout the day.","icon":"clear-day","data":[{"time":1422802800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.94,"apparentTemperature":1.07,"dewPoint":-2.09,"humidity":0.7,"windSpeed":1.91,"windBearing":295,"visibility":14.52,"pressure":1029.11},{"time":1422806400,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.65,"dewPoint":-2.12,"humidity":0.7,"windSpeed":2.23,"windBearing":292,"visibility":14.52,"pressure":1028.7},{"time":1422810000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.78,"dewPoint":-2.17,"humidity":0.69,"windSpeed":2.11,"windBearing":270,"visibility":14.16,"pressure":1028.52},{"time":1422813600,"summary":"Partly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.47,"dewPoint":-2.12,"humidity":0.7,"windSpeed":2.36,"windBearing":282,"visibility":15.77,"cloudCover":0.38,"pressure":1028.25},{"time":1422817200,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.65,"dewPoint":-2.13,"humidity":0.7,"windSpeed":2.19,"windBearing":285,"visibility":14.52,"pressure":1028.16},{"time":1422820800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.56,"dewPoint":-1.72,"humidity":0.72,"windSpeed":2.32,"windBearing":301,"visibility":12.41,"pressure":1028.21},{"time":1422824400,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.55,"dewPoint":-1.82,"humidity":0.71,"windSpeed":2.28,"windBearing":295,"visibility":14.95,"cloudCover":0.75,"pressure":1028.25},{"time":1422828000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.97,"apparentTemperature":0.69,"dewPoint":-2.1,"humidity":0.69,"windSpeed":2.29,"windBearing":304,"visibility":12.41,"pressure":1028.55},{"time":1422831600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":3.51,"apparentTemperature":1.37,"dewPoint":-2.53,"humidity":0.65,"windSpeed":2.24,"windBearing":294,"visibility":11.76,"pressure":1028.66},{"time":1422835200,"summary":"Mostly Cloudy","icon":"partly-cloudy-day","precipType":"rain","temperature":4.28,"apparentTemperature":2.64,"dewPoint":-3.14,"humidity":0.58,"windSpeed":1.89,"windBearing":285,"visibility":14.63,"cloudCover":0.74,"pressure":1028.55},{"time":1422838800,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":4.95,"apparentTemperature":2.61,"dewPoint":-3.67,"humidity":0.54,"windSpeed":2.79,"windBearing":301,"visibility":11.76,"pressure":1028.43},{"time":1422842400,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":6.13,"apparentTemperature":3.48,"dewPoint":-3.35,"humidity":0.51,"windSpeed":3.6,"windBearing":316,"visibility":11.76,"pressure":1027.97},{"time":1422846000,"summary":"Mostly Cloudy","icon":"partly-cloudy-day","precipType":"rain","temperature":5.99,"apparentTemperature":3.08,"dewPoint":-3.91,"humidity":0.49,"windSpeed":3.99,"windBearing":320,"visibility":14.45,"cloudCover":0.75,"pressure":1027.18},{"time":1422849600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":6.01,"apparentTemperature":3.01,"dewPoint":-3.74,"humidity":0.5,"windSpeed":4.17,"windBearing":329,"visibility":11.76,"pressure":1026.79},{"time":1422853200,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.94,"apparentTemperature":3.4,"dewPoint":-3.33,"humidity":0.51,"windSpeed":3.34,"windBearing":343,"visibility":11.76,"pressure":1026.69},{"time":1422856800,"summary":"Overcast","icon":"cloudy","precipType":"rain","temperature":5.78,"apparentTemperature":2.98,"dewPoint":-3.41,"humidity":0.52,"windSpeed":3.72,"windBearing":342,"visibility":14.71,"cloudCover":0.96,"pressure":1026.77},{"time":1422860400,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.74,"apparentTemperature":3.42,"dewPoint":-3.07,"humidity":0.53,"windSpeed":2.97,"windBearing":337,"visibility":11.72,"pressure":1026.87},{"time":1422864000,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.33,"apparentTemperature":2.86,"dewPoint":-2.67,"humidity":0.56,"windSpeed":3.07,"windBearing":329,"visibility":11.76,"pressure":1027.16},{"time":1422867600,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":4.43,"apparentTemperature":1.56,"dewPoint":-1.98,"humidity":0.63,"windSpeed":3.37,"windBearing":323,"visibility":14.45,"cloudCover":0.75,"pressure":1027.26},{"time":1422871200,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.36,"apparentTemperature":1.63,"dewPoint":-1.97,"humidity":0.63,"windSpeed":3.14,"windBearing":315,"visibility":11.76,"pressure":1027.55},{"time":1422874800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.25,"apparentTemperature":1.71,"dewPoint":-1.68,"humidity":0.65,"windSpeed":2.87,"windBearing":300,"visibility":12.41,"pressure":1027.77},{"time":1422878400,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":4.27,"apparentTemperature":1.95,"dewPoint":-1.74,"humidity":0.65,"windSpeed":2.59,"windBearing":313,"visibility":15.8,"cloudCover":0.76,"pressure":1027.98},{"time":1422882000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.25,"apparentTemperature":1.78,"dewPoint":-1.73,"humidity":0.65,"windSpeed":2.77,"windBearing":297,"visibility":14.52,"pressure":1028.02},{"time":1422885600,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.24,"apparentTemperature":2.03,"dewPoint":-1.72,"humidity":0.65,"windSpeed":2.46,"windBearing":296,"visibility":14.52,"pressure":1028.12}]},"daily":{"data":[{"time":1422802800,"summary":"Clear throughout the day.","icon":"clear-day","sunriseTime":1422827846,"sunsetTime":1422865697,"moonPhase":0.44,"precipType":"rain","temperatureMin":2.84,"temperatureMinTime":1422824400,"temperatureMax":6.13,"temperatureMaxTime":1422842400,"apparentTemperatureMin":0.47,"apparentTemperatureMinTime":1422813600,"apparentTemperatureMax":3.48,"apparentTemperatureMaxTime":1422842400,"dewPoint":-2.49,"humidity":0.62,"windSpeed":2.62,"windBearing":310,"visibility":13.45,"cloudCover":0.73,"pressure":1027.9}]},"flags":{"sources":["isd"],"isd-stations":["476490-99999","476510-99999","476630-99999","476840-99999","477500-99999","477590-99999","477610-99999","477690-99999","477700-99999","477710-99999","477720-99999","477740-99999","477760-99999","477770-99999","477790-99999","477800-99999"],"units":"si"}}]
WORKING_DIRECTORY = getcwd()
def to_csv(a, key, path = WORKING_DIRECTORY, file_name = "new_file.csv"):
full_path = path + '/' + file_name
# Only those two keys are supported for this piece of code
# More work and debugging for the other keys is needed
if key == "hourly" or key == "daily":
pd.DataFrame.from_dict(a[key]["data"]).to_csv(full_path)
else:
raise Exception("Those keys are not supported yet!")
for k in a:
to_csv(k, "hourly", file_name="hourly.csv")
to_csv(k, "daily", file_name="daily.csv" )
解决方案2:手动执行此操作:(不是一个好方法，但是您可以学习如何在不使用外部模块的情况下解决这类问题.)
Solution2: Doing the job manually: (Not a good way but you can learn how you can face this kind of problems without using an external modules.)
# This code will use OrderedDict
# In order to maintain the order of the dicts
from collections import OrderedDict
a = [{"latitude":34.700869,"longitude":135.503755,"timezone":"Asia/Tokyo","offset":9,"currently":{"time":1422831600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":3.51,"apparentTemperature":1.37,"dewPoint":-2.53,"humidity":0.65,"windSpeed":2.24,"windBearing":294,"visibility":11.76,"pressure":1028.66},"hourly":{"summary":"Clear throughout the day.","icon":"clear-day","data":[{"time":1422802800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.94,"apparentTemperature":1.07,"dewPoint":-2.09,"humidity":0.7,"windSpeed":1.91,"windBearing":295,"visibility":14.52,"pressure":1029.11},{"time":1422806400,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.65,"dewPoint":-2.12,"humidity":0.7,"windSpeed":2.23,"windBearing":292,"visibility":14.52,"pressure":1028.7},{"time":1422810000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.78,"dewPoint":-2.17,"humidity":0.69,"windSpeed":2.11,"windBearing":270,"visibility":14.16,"pressure":1028.52},{"time":1422813600,"summary":"Partly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.47,"dewPoint":-2.12,"humidity":0.7,"windSpeed":2.36,"windBearing":282,"visibility":15.77,"cloudCover":0.38,"pressure":1028.25},{"time":1422817200,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.65,"dewPoint":-2.13,"humidity":0.7,"windSpeed":2.19,"windBearing":285,"visibility":14.52,"pressure":1028.16},{"time":1422820800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.88,"apparentTemperature":0.56,"dewPoint":-1.72,"humidity":0.72,"windSpeed":2.32,"windBearing":301,"visibility":12.41,"pressure":1028.21},{"time":1422824400,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":2.84,"apparentTemperature":0.55,"dewPoint":-1.82,"humidity":0.71,"windSpeed":2.28,"windBearing":295,"visibility":14.95,"cloudCover":0.75,"pressure":1028.25},{"time":1422828000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":2.97,"apparentTemperature":0.69,"dewPoint":-2.1,"humidity":0.69,"windSpeed":2.29,"windBearing":304,"visibility":12.41,"pressure":1028.55},{"time":1422831600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":3.51,"apparentTemperature":1.37,"dewPoint":-2.53,"humidity":0.65,"windSpeed":2.24,"windBearing":294,"visibility":11.76,"pressure":1028.66},{"time":1422835200,"summary":"Mostly Cloudy","icon":"partly-cloudy-day","precipType":"rain","temperature":4.28,"apparentTemperature":2.64,"dewPoint":-3.14,"humidity":0.58,"windSpeed":1.89,"windBearing":285,"visibility":14.63,"cloudCover":0.74,"pressure":1028.55},{"time":1422838800,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":4.95,"apparentTemperature":2.61,"dewPoint":-3.67,"humidity":0.54,"windSpeed":2.79,"windBearing":301,"visibility":11.76,"pressure":1028.43},{"time":1422842400,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":6.13,"apparentTemperature":3.48,"dewPoint":-3.35,"humidity":0.51,"windSpeed":3.6,"windBearing":316,"visibility":11.76,"pressure":1027.97},{"time":1422846000,"summary":"Mostly Cloudy","icon":"partly-cloudy-day","precipType":"rain","temperature":5.99,"apparentTemperature":3.08,"dewPoint":-3.91,"humidity":0.49,"windSpeed":3.99,"windBearing":320,"visibility":14.45,"cloudCover":0.75,"pressure":1027.18},{"time":1422849600,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":6.01,"apparentTemperature":3.01,"dewPoint":-3.74,"humidity":0.5,"windSpeed":4.17,"windBearing":329,"visibility":11.76,"pressure":1026.79},{"time":1422853200,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.94,"apparentTemperature":3.4,"dewPoint":-3.33,"humidity":0.51,"windSpeed":3.34,"windBearing":343,"visibility":11.76,"pressure":1026.69},{"time":1422856800,"summary":"Overcast","icon":"cloudy","precipType":"rain","temperature":5.78,"apparentTemperature":2.98,"dewPoint":-3.41,"humidity":0.52,"windSpeed":3.72,"windBearing":342,"visibility":14.71,"cloudCover":0.96,"pressure":1026.77},{"time":1422860400,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.74,"apparentTemperature":3.42,"dewPoint":-3.07,"humidity":0.53,"windSpeed":2.97,"windBearing":337,"visibility":11.72,"pressure":1026.87},{"time":1422864000,"summary":"Clear","icon":"clear-day","precipType":"rain","temperature":5.33,"apparentTemperature":2.86,"dewPoint":-2.67,"humidity":0.56,"windSpeed":3.07,"windBearing":329,"visibility":11.76,"pressure":1027.16},{"time":1422867600,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":4.43,"apparentTemperature":1.56,"dewPoint":-1.98,"humidity":0.63,"windSpeed":3.37,"windBearing":323,"visibility":14.45,"cloudCover":0.75,"pressure":1027.26},{"time":1422871200,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.36,"apparentTemperature":1.63,"dewPoint":-1.97,"humidity":0.63,"windSpeed":3.14,"windBearing":315,"visibility":11.76,"pressure":1027.55},{"time":1422874800,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.25,"apparentTemperature":1.71,"dewPoint":-1.68,"humidity":0.65,"windSpeed":2.87,"windBearing":300,"visibility":12.41,"pressure":1027.77},{"time":1422878400,"summary":"Mostly Cloudy","icon":"partly-cloudy-night","precipType":"rain","temperature":4.27,"apparentTemperature":1.95,"dewPoint":-1.74,"humidity":0.65,"windSpeed":2.59,"windBearing":313,"visibility":15.8,"cloudCover":0.76,"pressure":1027.98},{"time":1422882000,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.25,"apparentTemperature":1.78,"dewPoint":-1.73,"humidity":0.65,"windSpeed":2.77,"windBearing":297,"visibility":14.52,"pressure":1028.02},{"time":1422885600,"summary":"Clear","icon":"clear-night","precipType":"rain","temperature":4.24,"apparentTemperature":2.03,"dewPoint":-1.72,"humidity":0.65,"windSpeed":2.46,"windBearing":296,"visibility":14.52,"pressure":1028.12}]},"daily":{"data":[{"time":1422802800,"summary":"Clear throughout the day.","icon":"clear-day","sunriseTime":1422827846,"sunsetTime":1422865697,"moonPhase":0.44,"precipType":"rain","temperatureMin":2.84,"temperatureMinTime":1422824400,"temperatureMax":6.13,"temperatureMaxTime":1422842400,"apparentTemperatureMin":0.47,"apparentTemperatureMinTime":1422813600,"apparentTemperatureMax":3.48,"apparentTemperatureMaxTime":1422842400,"dewPoint":-2.49,"humidity":0.62,"windSpeed":2.62,"windBearing":310,"visibility":13.45,"cloudCover":0.73,"pressure":1027.9}]},"flags":{"sources":["isd"],"isd-stations":["476490-99999","476510-99999","476630-99999","476840-99999","477500-99999","477590-99999","477610-99999","477690-99999","477700-99999","477710-99999","477720-99999","477740-99999","477760-99999","477770-99999","477790-99999","477800-99999"],"units":"si"}}]
def get_data(a, key):
# This part of code for the complex parts of the dict
# For: the key 'hourly' and 'daily'
data, final = [], ''
if key == 'daily' or key == 'hourly':
data = [OrderedDict(k) for k in a[key]["data"]]
# Note here that your JSON don't have the same numbers of keys in their inside dicts
# This is why this code fetch the max dict's items
# For example: The key 'cloudCover' isn't present in all the
# dicts with key 'hourly' of the original JSON
max_topic = max(map(lambda x: x.keys(), data), key=lambda x: len(x))
topic = ','.join(max_topic) + '\n'
for k in data:
for j in max_topic:
try:
final += str(k[j]) +', '
except KeyError:
# if the keys isn't found add an empty string
final += ' ,'
final += '\n'
# This part for the key 'flags'
elif key == 'flags':
data = a[key]
topic = "{0},{1},{2}\n".format("isd-stations", "sources", "units")
sub_final = ',{0},{1}\n'.format(''.join(a[key]['sources']), a[key]['units'])
final = sub_final + ',\n'.join(a[key]["isd-stations"])
elif key == 'currently':
data = OrderedDict(a[key])
topic = ','.join(a[key].keys()) + '\n'
final = ','.join(map(str, a[key].values()))
# This part for the other keys like: 'offset'
# If there will be another key
# This code will throw an exception 'KeyError'
else:
topic = key + '\n'
final = str(a[key]) + '\n'
return topic, final
def write_to_csv(a, key, file_name="new_file.csv"):
topic, data = get_data(a, key)
with open(file_name, 'a') as f:
f.write("{0}{1}".format(topic, ''.join(data)))
# An empty line used if you will run this code multiple times
# and you'll append your data in a single file
# This will seperate your formated data for better visualisation
f.write("\n")
for k in a:
write_to_csv(k, "hourly", "hourly.csv")
write_to_csv(k, "daily", "daily.csv")
write_to_csv(k, "flags", "flags.csv")
输出:
hourly.csv
temperature,windBearing,cloudCover,icon,precipType,summary,humidity,dewPoint,time,pressure,visibility,windSpeed,apparentTemperature
2.94, 295, ,clear-night, rain, Clear, 0.7, -2.09, 1422802800, 1029.11, 14.52, 1.91, 1.07,
2.88, 292, ,clear-night, rain, Clear, 0.7, -2.12, 1422806400, 1028.7, 14.52, 2.23, 0.65,
2.88, 270, ,clear-night, rain, Clear, 0.69, -2.17, 1422810000, 1028.52, 14.16, 2.11, 0.78,
2.84, 282, 0.38, partly-cloudy-night, rain, Partly Cloudy, 0.7, -2.12, 1422813600, 1028.25, 15.77, 2.36, 0.47,
...
daily.csv
windBearing,apparentTemperatureMax,summary,apparentTemperatureMin,apparentTemperatureMaxTime,temperatureMaxTime,apparentTemperatureMinTime,windSpeed,pressure,temperatureMin,sunsetTime,icon,cloudCover,moonPhase,time,temperatureMinTime,visibility,dewPoint,sunriseTime,precipType,humidity,temperatureMax
310, 3.48, Clear throughout the day., 0.47, 1422842400, 1422842400, 1422813600, 2.62, 1027.9, 2.84, 1422865697, clear-day, 0.73, 0.44, 1422802800, 1422824400, 13.45, -2.49, 1422827846, rain, 0.62, 6.13,
...
flags.csv
isd-stations,sources,units
,isd,si
476490-99999,
476510-99999,
476630-99999,
476840-99999,
...
这篇关于Python抓取JSON转换为CSV的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
