从深度嵌套的JSON创建Pandas DataFrame [英] Create a Pandas DataFrame from deeply nested JSON
问题描述
我正在尝试从深度嵌套的JSON字符串创建单个Pandas DataFrame对象.
I'm trying to create a single Pandas DataFrame object from a deeply nested JSON string.
JSON模式为:
{"intervals": [
{
pivots: "Jane Smith",
"series": [
{
"interval_id": 0,
"p_value": 1
},
{
"interval_id": 1,
"p_value": 1.1162791357932633e-8
},
{
"interval_id": 2,
"p_value": 0.0000028675012051504467
}
],
},
{
"pivots": "Bob Smith",
"series": [
{
"interval_id": 0,
"p_value": 1
},
{
"interval_id": 1,
"p_value": 1.1162791357932633e-8
},
{
"interval_id": 2,
"p_value": 0.0000028675012051504467
}
]
}
]
}
期望的结果，我需要将其展平以生成表格:
Desired Outcome I need to flatten this to produce a table:
Actor Interval_id Interval_id Interval_id ...
Jane Smith 1 1.1162 0.00000 ...
Bob Smith 1 1.1162 0.00000 ...
第一列是Pivots值，其余列是存储在列表series中的键interval_id和p_value的值.
The first column is the Pivots values, and the remaining columns are the values of the keys interval_id and p_value stored in the list series.
到目前为止，我已经
import requests as r
import pandas as pd
actor_data = r.get("url/to/data").json['data']['intervals']
df = pd.DataFrame(actor_data)
actor_data是一个列表，其中长度等于个人数，即pivots.values(). df对象只是返回
actor_data is a list where the length is equal to the number of individuals ie pivots.values(). The df object simply returns
<bound method DataFrame.describe of pivots Series
0 Jane Smith [{u'p_value': 1.0, u'interval_id': 0}, {u'p_va...
1 Bob Smith [{u'p_value': 1.0, u'interval_id': 0}, {u'p_va...
.
.
.
如何遍历series列表以获取dict值并创建N个不同的列?我是否应该尝试为series列表创建一个DataFrame，重塑它的形状，然后用角色名称绑定列?
How can I iterate through that series list to get to the dict values and create N distinct columns? Should I try to create a DataFrame for the series list, reshape it,and then do a column bind with the actor names?
更新:
pvalue_list = [i['p_value'] for i in json_data['series']]
这给了我一个列表清单.现在，我需要弄清楚如何将每个列表添加为DataFrame中的一行.
this gives me a list of lists. Now I need to figure out how to add each list as a row in a DataFrame.
value_list = []
for i in pvalue_list:
pvs = [j['p_value'] for j in i]
value_list = value_list.append(pvs)
return value_list
这将返回NoneType
This returns a NoneType
解决方案
def get_hypthesis_data():
raw_data = r.get("/url/to/data").json()['data']
actor_dict = {}
for actor_series in raw_data['intervals']:
actor = actor_series['pivots']
p_values = []
for interval in actor_series['series']:
p_values.append(interval['p_value'])
actor_dict[actor] = p_values
return pd.DataFrame(actor_dict).T
这将返回正确的DataFrame.我对它进行了移调，因此个人是行而不是列.
This returns the correct DataFrame. I transposed it so the individuals were rows and not columns.
推荐答案
我认为，以产生重复列名的方式来组织数据只会在以后给您带来麻烦.恕我直言，更好的方法是为pivots，interval_id和p_value中的每一个创建一个列.将数据加载到熊猫中后，这将使查询数据变得非常容易.
I think organizing your data in way that yields repeating column names is only going to create headaches for you later on down the road. A better approach IMHO is to create a column for each of pivots, interval_id, and p_value. This will make extremely easy to query your data after loading it into pandas.
此外，您的JSON中有一些错误.我通过此运行了它，以查找错误.
Also, your JSON has some errors in it. I ran it through this to find the errors.
jq 在这里有帮助
jq helps here
import sh
jq = sh.jq.bake('-M') # disable colorizing
json_data = "from above"
rule = """[{pivots: .intervals[].pivots,
interval_id: .intervals[].series[].interval_id,
p_value: .intervals[].series[].p_value}]"""
out = jq(rule, _in=json_data).stdout
res = pd.DataFrame(json.loads(out))
这将产生类似于
interval_id p_value pivots
32 2 2.867501e-06 Jane Smith
33 2 1.000000e+00 Jane Smith
34 2 1.116279e-08 Jane Smith
35 2 2.867501e-06 Jane Smith
36 0 1.000000e+00 Bob Smith
37 0 1.116279e-08 Bob Smith
38 0 2.867501e-06 Bob Smith
39 0 1.000000e+00 Bob Smith
40 0 1.116279e-08 Bob Smith
41 0 2.867501e-06 Bob Smith
42 1 1.000000e+00 Bob Smith
43 1 1.116279e-08 Bob Smith
改编自此评论
当然，您始终可以调用res.drop_duplicates()来删除重复的行.这给出了
Of course, you can always call res.drop_duplicates() to remove the duplicate rows. This gives
In [175]: res.drop_duplicates()
Out[175]:
interval_id p_value pivots
0 0 1.000000e+00 Jane Smith
1 0 1.116279e-08 Jane Smith
2 0 2.867501e-06 Jane Smith
6 1 1.000000e+00 Jane Smith
7 1 1.116279e-08 Jane Smith
8 1 2.867501e-06 Jane Smith
12 2 1.000000e+00 Jane Smith
13 2 1.116279e-08 Jane Smith
14 2 2.867501e-06 Jane Smith
36 0 1.000000e+00 Bob Smith
37 0 1.116279e-08 Bob Smith
38 0 2.867501e-06 Bob Smith
42 1 1.000000e+00 Bob Smith
43 1 1.116279e-08 Bob Smith
44 1 2.867501e-06 Bob Smith
48 2 1.000000e+00 Bob Smith
49 2 1.116279e-08 Bob Smith
50 2 2.867501e-06 Bob Smith
[18 rows x 3 columns]
这篇关于从深度嵌套的JSON创建Pandas DataFrame的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
