通过使用切片列表从DataFrame获取行 [英] Get rows from a DataFrame by using a list of slices
问题描述
我有几百万行数据框,并且需要从中选择感兴趣的部分.我正在寻找一种高效(可能最快的方式)的方法.
I have a several million row data frame, and a list of interesting sections I need to select out of it. I'm looking for a highly efficient (read as: fastest possible) way of doing this.
我知道我可以做到:
slices = [slice(0,10), slice(20,50), slice(1000,5000)]
for slice in slices:
df.loc[slice, 'somecolumn'] = True
...但是这似乎是完成工作的一种低效方式. 真的很慢.
... but that just seems like an inefficient way of getting the job done. It's really slow.
这似乎比上面的for循环快,但是我不确定这是否是最好的方法:
This seems faster than the for loop above, but I'm not sure if this is the best possible approach:
from itertools import chain
ranges = chain.from_iterable(slices)
df.loc[ranges, 'somecolumns'] = True
这似乎也不可行,即使它应该也不会起作用
This also doesn't work, even though it seems that maybe it should:
df.loc[slices, 'somecolumns'] = True
TypeError: unhashable type: 'slice'
我主要关注的是性能.由于要处理的数据帧的大小,我需要最好的方法.
My primary concern in this is performance. I need the best I can get out of this due to the size of the data frames I am dealing with.
推荐答案
熊猫
您可以尝试一些技巧:
pandas
You can try a couple of tricks:
- 使用
np.r_
来连接将对象放入单个NumPy数组.使用NumPy数组建立索引通常是有效的,因为它们在Pandas框架内部使用. - 通过
pd.DataFrame.iloc
使用位置整数索引a>而不是主要基于标签的loc
一个>.前者的限制更严格,并且与NumPy索引更加一致.
这是一个演示:
# some example dataframe
df = pd.DataFrame(dict(zip('ABCD', np.arange(100).reshape((4, 25)))))
# concatenate multiple slices
slices = np.r_[slice(0, 3), slice(6, 10), slice(15, 20)]
# use integer indexing
df.iloc[slices, df.columns.get_loc('C')] = 0
numpy
如果系列存储在连续的内存块中(数字(或布尔)数组通常是这种情况),则可以尝试就地更新基础NumPy数组.首先按照上述方法通过np.r_
定义slices
,然后使用:
numpy
If your series is held in a contiguous memory block, which is usually the case with numeric (or Boolean) arrays, you can try updating the underlying NumPy array in-place. First define slices
via np.r_
as above, then use:
df['C'].values[slices] = 0
这会绕过Pandas界面以及通过常规索引方法进行的所有相关检查.
This by-passes the Pandas interface and any associated checks which occur via the regular indexing methods.
这篇关于通过使用切片列表从DataFrame获取行的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!