在pandas DataFrame列中存储多维数组 [英] Storing multidimensional arrays in pandas DataFrame columns
问题描述
我希望将熊猫用作主要的Trace(MCMC参数空间中的一系列点)对象.
我有一个string-> array字典的列表，我想将它们存储在熊猫中.字典中的键始终相同，并且对于每个键，numpy数组的形状始终相同，但是不同键的形状可能不同，并且维数可能不同.
我一直在使用self.append(dict_list, ignore_index = True)，它对于1d值似乎很好用，但是对于nd> 1值，pandas将这些值存储为对象，因此无法进行漂亮的绘图和其他处理.关于如何获得更好的行为有什么建议吗?
样本数据
point = {'x': array(-0.47652306228698005),
'y': array([[-0.41809043],
[ 0.48407823]])}
points = 10 * [ point]
我希望能够做
df = DataFrame(points)
或
df = DataFrame()
df.append(points, ignore_index=True)
并拥有
>> df['x'][1].shape
()
>> df['y'][1].shape
(2,1)
相对较新的库 xray [1]具有完全符合您要求的Dataset和DataArray结构. /p>
这是我对您的问题的看法，写为 IPython 会话:
>>> import numpy as np
>>> import xray
>>> ## Prepare data:
>>> #
>>> point = {'x': np.array(-0.47652306228698005),
... 'y': np.array([[-0.41809043],
... [ 0.48407823]])}
>>> points = 10 * [point]
>>> ## Convert to Xray DataArrays:
>>> #
>>> list_x = [p['x'] for p in points]
>>> list_y = [p['y'] for p in points]
>>> da_x = xray.DataArray(list_x, [('x', range(len(list_x)))])
>>> da_y = xray.DataArray(list_y, [
... ('x', range(len(list_y))),
... ('y0', range(2)),
... ('y1', [0]),
... ])
以下是我们到目前为止构建的两个DataArray实例:
>>> print(da_x)
<xray.DataArray (x: 10)>
array([-0.47652306, -0.47652306, -0.47652306, -0.47652306, -0.47652306,
-0.47652306, -0.47652306, -0.47652306, -0.47652306, -0.47652306])
Coordinates:
* x (x) int32 0 1 2 3 4 5 6 7 8 9
>>> print(da_y.T) ## Transposed, to save lines.
<xray.DataArray (y1: 1, y0: 2, x: 10)>
array([[[-0.41809043, -0.41809043, -0.41809043, -0.41809043, -0.41809043,
-0.41809043, -0.41809043, -0.41809043, -0.41809043, -0.41809043],
[ 0.48407823, 0.48407823, 0.48407823, 0.48407823, 0.48407823,
0.48407823, 0.48407823, 0.48407823, 0.48407823, 0.48407823]]])
Coordinates:
* x (x) int32 0 1 2 3 4 5 6 7 8 9
* y0 (y0) int32 0 1
* y1 (y1) int32 0
我们现在可以将这两个DataArray在其共同的x维度上合并为DataSet:
>>> ds = xray.Dataset({'X':da_x, 'Y':da_y})
>>> print(ds)
<xray.Dataset>
Dimensions: (x: 10, y0: 2, y1: 1)
Coordinates:
* x (x) int32 0 1 2 3 4 5 6 7 8 9
* y0 (y0) int32 0 1
* y1 (y1) int32 0
Data variables:
X (x) float64 -0.4765 -0.4765 -0.4765 -0.4765 -0.4765 -0.4765 -0.4765 ...
Y (x, y0, y1) float64 -0.4181 0.4841 -0.4181 0.4841 -0.4181 0.4841 -0.4181 ...
我们最终可以按照您想要的方式访问和聚合数据:
>>> ds['X'].sum()
<xray.DataArray 'X' ()>
array(-4.765230622869801)
>>> ds['Y'].sum()
<xray.DataArray 'Y' ()>
array(0.659878)
>>> ds['Y'].sum(axis=1)
<xray.DataArray 'Y' (x: 10, y1: 1)>
array([[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878]])
Coordinates:
* x (x) int32 0 1 2 3 4 5 6 7 8 9
* y1 (y1) int32 0
>>> np.all(ds['Y'].sum(axis=1) == ds['Y'].sum(dim='y0'))
True
>>>> ds['X'].sum(dim='y0')
Traceback (most recent call last):
ValueError: 'y0' not found in array dimensions ('x',)
[1]一个用于处理带有标签的N维数据的库，就像熊猫对2D一样: http://xray.readthedocs.org/en/stable/data-structures.html#dataset
I'm hoping to use pandas as the main Trace (series of points in parameter space from MCMC) object.
I have a list of dicts of string->array which I would like to store in pandas. The keys in the dicts are always the same, and for each key the shape of the numpy array is always the same, but the shape may be different for different keys and could have a different number of dimensions.
I had been using self.append(dict_list, ignore_index = True) which seems to work well for 1d values, but for nd>1 values pandas stores the values as objects which doesn't allow for nice plotting and other nice things. Any suggestions on how to get better behavior?
Sample data
point = {'x': array(-0.47652306228698005),
'y': array([[-0.41809043],
[ 0.48407823]])}
points = 10 * [ point]
I'd like to be able to do something like
df = DataFrame(points)
or
df = DataFrame()
df.append(points, ignore_index=True)
and have
>> df['x'][1].shape
()
>> df['y'][1].shape
(2,1)
The relatively-new library xray[1] has Dataset and DataArray structures that do exactly what you ask.
Here it is my take on your problem, written as an IPython session:
>>> import numpy as np
>>> import xray
>>> ## Prepare data:
>>> #
>>> point = {'x': np.array(-0.47652306228698005),
... 'y': np.array([[-0.41809043],
... [ 0.48407823]])}
>>> points = 10 * [point]
>>> ## Convert to Xray DataArrays:
>>> #
>>> list_x = [p['x'] for p in points]
>>> list_y = [p['y'] for p in points]
>>> da_x = xray.DataArray(list_x, [('x', range(len(list_x)))])
>>> da_y = xray.DataArray(list_y, [
... ('x', range(len(list_y))),
... ('y0', range(2)),
... ('y1', [0]),
... ])
These are the two DataArray instances we built so far:
>>> print(da_x)
<xray.DataArray (x: 10)>
array([-0.47652306, -0.47652306, -0.47652306, -0.47652306, -0.47652306,
-0.47652306, -0.47652306, -0.47652306, -0.47652306, -0.47652306])
Coordinates:
* x (x) int32 0 1 2 3 4 5 6 7 8 9
>>> print(da_y.T) ## Transposed, to save lines.
<xray.DataArray (y1: 1, y0: 2, x: 10)>
array([[[-0.41809043, -0.41809043, -0.41809043, -0.41809043, -0.41809043,
-0.41809043, -0.41809043, -0.41809043, -0.41809043, -0.41809043],
[ 0.48407823, 0.48407823, 0.48407823, 0.48407823, 0.48407823,
0.48407823, 0.48407823, 0.48407823, 0.48407823, 0.48407823]]])
Coordinates:
* x (x) int32 0 1 2 3 4 5 6 7 8 9
* y0 (y0) int32 0 1
* y1 (y1) int32 0
We can now merge these two DataArray on their common x dimension into a DataSet:
>>> ds = xray.Dataset({'X':da_x, 'Y':da_y})
>>> print(ds)
<xray.Dataset>
Dimensions: (x: 10, y0: 2, y1: 1)
Coordinates:
* x (x) int32 0 1 2 3 4 5 6 7 8 9
* y0 (y0) int32 0 1
* y1 (y1) int32 0
Data variables:
X (x) float64 -0.4765 -0.4765 -0.4765 -0.4765 -0.4765 -0.4765 -0.4765 ...
Y (x, y0, y1) float64 -0.4181 0.4841 -0.4181 0.4841 -0.4181 0.4841 -0.4181 ...
And we can finally access and aggregate data the way you wanted:
>>> ds['X'].sum()
<xray.DataArray 'X' ()>
array(-4.765230622869801)
>>> ds['Y'].sum()
<xray.DataArray 'Y' ()>
array(0.659878)
>>> ds['Y'].sum(axis=1)
<xray.DataArray 'Y' (x: 10, y1: 1)>
array([[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878],
[ 0.0659878]])
Coordinates:
* x (x) int32 0 1 2 3 4 5 6 7 8 9
* y1 (y1) int32 0
>>> np.all(ds['Y'].sum(axis=1) == ds['Y'].sum(dim='y0'))
True
>>>> ds['X'].sum(dim='y0')
Traceback (most recent call last):
ValueError: 'y0' not found in array dimensions ('x',)
[1] A library for handling N-dimensional data with labels, like pandas does for 2D: http://xray.readthedocs.org/en/stable/data-structures.html#dataset
这篇关于在pandas DataFrame列中存储多维数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
