pandas 面板中的布尔面罩 [英] boolean mask in pandas panel
问题描述
我在掩盖面板时遇到了一些麻烦，就像在使用DataFrame一样.我想做的事情很简单，但是我没有找到查看文档和在线论坛的方法.我在下面有一个简单的例子:
i am having some trouble masking a panel in the same way that I would a DataFrame. What I want to do feels simple, but I have not found a way looking at the docs and online forums. I have a simple example below:
import pandas
import numpy as np
import datetime
start_date = datetime.datetime(2009,3,1,6,29,59)
r = pandas.date_range(start_date, periods=12)
cols_1 = ['AAPL', 'AAPL', 'GOOG', 'GOOG', 'GS', 'GS']
cols_2 = ['close', 'rate', 'close', 'rate', 'close', 'rate']
dat = np.random.randn(12, 6)
dftst = pandas.DataFrame(dat, columns=pandas.MultiIndex.from_arrays([cols_1, cols_2], names=['ticker','field']), index=r)
pn = dftst.T.to_panel().transpose(2,0,1)
print pn
Out[14]:
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 12 (major_axis) x 3 (minor_axis)
Items axis: close to rate
Major_axis axis: 2009-03-01 06:29:59 to 2009-03-12 06:29:59
Minor_axis axis: AAPL to GS
我现在有了一个Panel对象，如果我沿着项轴进行切片，就会得到一个DataFrame
I now have a Panel object, if I take a slice along the items axis, I get a DataFrame
close_p = pn['close']
print close_p
Out[16]:
ticker AAPL GOOG GS
2009-03-01 06:29:59 -0.082203 -0.286354 1.227193
2009-03-02 06:29:59 0.340005 -0.688933 -1.505137
2009-03-03 06:29:59 -0.525567 0.321858 -0.035047
2009-03-04 06:29:59 -0.123549 -0.841781 -0.616523
2009-03-05 06:29:59 -0.407504 0.188372 1.311262
2009-03-06 06:29:59 0.272883 0.817179 0.584664
2009-03-07 06:29:59 -1.767227 1.168876 0.443096
2009-03-08 06:29:59 -0.685501 -0.534373 -0.063906
2009-03-09 06:29:59 0.851820 0.068740 0.566537
2009-03-10 06:29:59 0.390678 -0.012422 -0.152375
2009-03-11 06:29:59 -0.985585 -0.917705 -0.585091
2009-03-12 06:29:59 0.067498 -0.764343 0.497270
我可以通过两种方式过滤此数据:
I can filter this data in two ways:
1)我创建一个掩码，并如下掩码数据:
1) I create a mask and mask the data as follows:
msk = close_p > 0
close_p = close_p.mask(msk)
2)我可以通过上面的msk中的布尔运算符进行切片
2) I can just slice by the boolean operator in msk above
close_p = close_p[close_p > 0]
Out[28]:
ticker AAPL GOOG GS
2009-03-01 06:29:59 NaN NaN 1.227193
2009-03-02 06:29:59 0.340005 NaN NaN
2009-03-03 06:29:59 NaN 0.321858 NaN
2009-03-04 06:29:59 NaN NaN NaN
2009-03-05 06:29:59 NaN 0.188372 1.311262
2009-03-06 06:29:59 0.272883 0.817179 0.584664
2009-03-07 06:29:59 NaN 1.168876 0.443096
2009-03-08 06:29:59 NaN NaN NaN
2009-03-09 06:29:59 0.851820 0.068740 0.566537
2009-03-10 06:29:59 0.390678 NaN NaN
2009-03-11 06:29:59 NaN NaN NaN
2009-03-12 06:29:59 0.067498 NaN 0.497270
我不知道该怎么做，是基于一个没有for循环的掩码过滤所有数据.我可以执行以下操作:
What I cannot figure out how to do is filter all of my data based on a mask without a for loop. I can do the following:
msk = (pn['rate'] > 0) & (pn['close'] > 0)
def mask_panel(pan, msk):
for item in pan.items:
pan[item] = pan[item].mask(msk)
return pan
print pn['close']
Out[32]:
ticker AAPL GOOG GS
2009-03-01 06:29:59 -0.082203 -0.286354 1.227193
2009-03-02 06:29:59 0.340005 -0.688933 -1.505137
2009-03-03 06:29:59 -0.525567 0.321858 -0.035047
2009-03-04 06:29:59 -0.123549 -0.841781 -0.616523
2009-03-05 06:29:59 -0.407504 0.188372 1.311262
2009-03-06 06:29:59 0.272883 0.817179 0.584664
2009-03-07 06:29:59 -1.767227 1.168876 0.443096
2009-03-08 06:29:59 -0.685501 -0.534373 -0.063906
2009-03-09 06:29:59 0.851820 0.068740 0.566537
2009-03-10 06:29:59 0.390678 -0.012422 -0.152375
2009-03-11 06:29:59 -0.985585 -0.917705 -0.585091
2009-03-12 06:29:59 0.067498 -0.764343 0.497270
mask_panel(pn, msk)
print pn['close']
Out[34]:
ticker AAPL GOOG GS
2009-03-01 06:29:59 -0.082203 -0.286354 NaN
2009-03-02 06:29:59 NaN -0.688933 -1.505137
2009-03-03 06:29:59 -0.525567 NaN -0.035047
2009-03-04 06:29:59 -0.123549 -0.841781 -0.616523
2009-03-05 06:29:59 -0.407504 NaN NaN
2009-03-06 06:29:59 NaN NaN NaN
2009-03-07 06:29:59 -1.767227 NaN NaN
2009-03-08 06:29:59 -0.685501 -0.534373 -0.063906
2009-03-09 06:29:59 NaN NaN NaN
2009-03-10 06:29:59 NaN -0.012422 -0.152375
2009-03-11 06:29:59 -0.985585 -0.917705 -0.585091
2009-03-12 06:29:59 NaN -0.764343 NaN
因此上述循环可以解决问题.我知道有一种使用ndarray进行此操作的更快的矢量化方法，但我尚未将其组合在一起.看来这应该是内置在pandas库中的功能.如果我没有办法做到这一点，那么任何建议将不胜感激.
So the above loop does the trick. I know there is a faster vectorized way of doing this using the ndarray, but I have not put that together yet. It also seems like this should be functionality that is built into the pandas library. If there is a way to do this that I am missing, any suggestions would be much appreciated.
推荐答案
我认为这可以解决(以及Panel.where应该做什么)，但这有点不平凡，因为它 必须处理很多案件)
I think this will work (and what Panel.where should do, but its a bit non-trivial because it has to handle a bunch of cases)
# construct the mask in 2-d (a frame)
In [36]: mask = (pn['close']>0) & (pn['rate']>0)
In [37]: mask
Out[37]:
ticker AAPL GOOG GS
2009-03-01 06:29:59 False False False
2009-03-02 06:29:59 False False True
....
# here's the key, this broadcasts, setting the values which
# don't meet the condition to nan
In [38]: masked_values = np.where(mask,pn.values,np.nan)
# reconstruct the panel (the _construct_axes_dict is an internal function that returns
# dict of the axes, e.g. items -> the items, major_axis -> .....
In [42]: x = pd.Panel(masked_values,**pn._construct_axes_dict())
Out[42]:
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 12 (major_axis) x 3 (minor_axis)
Items axis: close to rate
Major_axis axis: 2009-03-01 06:29:59 to 2009-03-12 06:29:59
Minor_axis axis: AAPL to GS
# the values
In [43]: x
Out[43]:
array([[[ nan, nan, nan],
[ nan, nan, 0.09575723],
[ nan, nan, nan],
[ nan, nan, nan],
[ nan, 2.07229823, 0.04347515],
[ nan, nan, nan],
[ nan, nan, 2.18342239],
[ nan, nan, 1.73674381],
[ nan, 2.01173087, nan],
[ 0.24109645, 0.94583072, nan],
[ 0.36953467, nan, 0.18044432],
[ 1.74164222, 1.02314752, 1.73736033]],
[[ nan, nan, nan],
[ nan, nan, 0.06960387],
[ nan, nan, nan],
[ nan, nan, nan],
[ nan, 0.63202199, 0.56724391],
[ nan, nan, nan],
[ nan, nan, 0.71964824],
[ nan, nan, 1.03482927],
[ nan, 0.18256148, nan],
[ 1.29451667, 0.49804327, nan],
[ 2.04726538, nan, 0.12883128],
[ 0.70647885, 0.7277734 , 0.77844475]]])
这篇关于 pandas 面板中的布尔面罩的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
