通过使用切片列表从DataFrame获取行 [英] Get rows from a DataFrame by using a list of slices

问题描述

我有几百万行数据框，并且需要从中选择感兴趣的部分.我正在寻找一种高效(可能最快的方式)的方法.

I have a several million row data frame, and a list of interesting sections I need to select out of it. I'm looking for a highly efficient (read as: fastest possible) way of doing this.

我知道我可以做到:

slices = [slice(0,10), slice(20,50), slice(1000,5000)]
for slice in slices:
  df.loc[slice, 'somecolumn'] = True

...但是这似乎是完成工作的一种低效方式. 真的很慢.

... but that just seems like an inefficient way of getting the job done. It's really slow.

这似乎比上面的for循环快，但是我不确定这是否是最好的方法:

This seems faster than the for loop above, but I'm not sure if this is the best possible approach:

from itertools import chain
ranges = chain.from_iterable(slices)
df.loc[ranges, 'somecolumns'] = True

这似乎也不可行，即使它应该也不会起作用

This also doesn't work, even though it seems that maybe it should:

df.loc[slices, 'somecolumns'] = True

TypeError: unhashable type: 'slice'

我主要关注的是性能.由于要处理的数据帧的大小，我需要最好的方法.

My primary concern in this is performance. I need the best I can get out of this due to the size of the data frames I am dealing with.

推荐答案

熊猫

您可以尝试一些技巧:

pandas

You can try a couple of tricks:

1. 使用 np.r_ 来连接将对象放入单个NumPy数组.使用NumPy数组建立索引通常是有效的，因为它们在Pandas框架内部使用.
2. 通过 pd.DataFrame.iloc 使用位置整数索引a>而不是主要基于标签的 loc .前者的限制更严格，并且与NumPy索引更加一致.

这是一个演示:

# some example dataframe
df = pd.DataFrame(dict(zip('ABCD', np.arange(100).reshape((4, 25)))))

# concatenate multiple slices
slices = np.r_[slice(0, 3), slice(6, 10), slice(15, 20)]

# use integer indexing
df.iloc[slices, df.columns.get_loc('C')] = 0

numpy

如果系列存储在连续的内存块中(数字(或布尔)数组通常是这种情况)，则可以尝试就地更新基础NumPy数组.首先按照上述方法通过np.r_定义slices，然后使用:

numpy

If your series is held in a contiguous memory block, which is usually the case with numeric (or Boolean) arrays, you can try updating the underlying NumPy array in-place. First define slices via np.r_ as above, then use:

df['C'].values[slices] = 0

这会绕过Pandas界面以及通过常规索引方法进行的所有相关检查.

This by-passes the Pandas interface and any associated checks which occur via the regular indexing methods.

这篇关于通过使用切片列表从DataFrame获取行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！