有什么方法可以高效保存和读取多维数据吗? [英] Is there any way to save and read multi-dimension data with efficiency?
问题描述
-
我有一堆数据序列，包含1000个站点，每个站点都具有4个功能(例如温度，风，CO2浓度，太阳辐射).
I have a bunch of data series with 1000 stations and each station all have 4 features (e.g Temperature, Wind, CO2 concentration, solar radiation).
所有功能均按时间顺序排列，并具有小时分辨率.
All the features are in time-series with hourly resolution.
在熊猫的支持下，我在 .csv 文件中读取了此数据.
I read this data in .csv files with the support of Pandas.
现在，我需要将它们保存和组织在一起，以便更好地重复使用.
Now I need to save and organize them together for better re-use.
我创建了以"sample_x，feature_y"为标题的列.并且每一列都包含sample_x的feature_y的时间序列数据.
I creat columns entitled by 'sample_x, feature_y'. And each column contain the time series data of feature_y for sample_x.
此方法可行，但效率不高.因为我必须创建具有长列名的4000列.
This method is doable but not show efficiency. Because I had to creat like 4000 columns with long column name.
有什么更好的方法可以在Python中保存多维数据.我想要一个简单的解决方案，可以帮助我直接评估和处理特定数据.
Is there any better way to save multi-demensions data in Python. I want a simple solution that can help me assessing and handling with specific data directly.
感谢任何建议或解决方案！
Any advices or solution is appreciated!
推荐答案
我认为您可以使用 Panel ，然后根据需要将数据保存到 hdf5 .
I think you can use MultiIndex or Panel and then if necessary save data to hdf5.
还函数 concat 具有参数keys从list of DataFrames创建MultiIndex.
示例:
df1 = pd.DataFrame({'A':[1,2,3],
'B':[4,5,6],
'C':[7,8,9],
'D':[1,3,5]})
print (df1)
A B C D
0 1 4 7 1
1 2 5 8 3
2 3 6 9 5
df2 = df1 * 10
dfs = [df1, df2]
df3 = pd.concat(dfs, keys=['a','b'])
print (df3)
A B C D
a 0 1 4 7 1
1 2 5 8 3
2 3 6 9 5
b 0 10 40 70 10
1 20 50 80 30
2 30 60 90 50
print (df3.index)
MultiIndex(levels=[['a', 'b'], [0, 1, 2]],
labels=[[0, 0, 0, 1, 1, 1], [0, 1, 2, 0, 1, 2]])
wp = pd.Panel({'a' : df1, 'b' : df2})
print (wp)
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 3 (major_axis) x 4 (minor_axis)
Items axis: a to b
Major_axis axis: 0 to 2
Minor_axis axis: A to D
这篇关于有什么方法可以高效保存和读取多维数据吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
