大 pandas 中非唯一索引的性能影响是什么? [英] What is the performance impact of non-unique indexes in pandas?
问题描述
从熊猫文档中,我收集到了这些独特价值指数使得某些操作有效,并且偶尔会容忍非唯一指数。
From the pandas documentation, I've gathered that unique-valued indices make certain operations efficient, and that non-unique indices are occasionally tolerated.
来自外部,它看起来不像任何方式利用非独特的指数。例如,以下 ix
查询速度足够慢,似乎正在扫描整个数据帧
From the outside, it doesn't look like non-unique indices are taken advantage of in any way. For example, the following ix
query is slow enough that it seems to be scanning the entire dataframe
In [23]: import numpy as np
In [24]: import pandas as pd
In [25]: x = np.random.randint(0, 10**7, 10**7)
In [26]: df1 = pd.DataFrame({'x':x})
In [27]: df2 = df1.set_index('x', drop=False)
In [28]: %timeit df2.ix[0]
1 loops, best of 3: 402 ms per loop
In [29]: %timeit df1.ix[0]
10000 loops, best of 3: 123 us per loop
(我意识到两个 ix
查询不会返回相同的内容 - 这只是一个调用 ix
的示例唯一索引显得慢得多)
(I realize the two ix
queries don't return the same thing -- it's just an example that calls to ix
on a non-unique index appear much slower)
有没有办法让pandas在非唯一和/或排序索引上使用更快的查找方法,如二元搜索?
Is there any way to coax pandas into using faster lookup methods like binary search on non-unique and/or sorted indices?
推荐答案
当index是唯一的时,pandas使用哈希表将键映射到值O(1)。当索引是非唯一且已排序时,pandas使用二进制搜索O(logN),当索引是随机排序时,pandas需要检查索引O(N)中的所有键。
When index is unique, pandas use a hashtable to map key to value O(1). When index is non-unique and sorted, pandas use binary search O(logN), when index is random ordered pandas need to check all the keys in the index O(N).
你可以调用 sort_index
方法:
import numpy as np
import pandas as pd
x = np.random.randint(0, 200, 10**6)
df1 = pd.DataFrame({'x':x})
df2 = df1.set_index('x', drop=False)
df3 = df2.sort_index()
%timeit df1.loc[100]
%timeit df2.loc[100]
%timeit df3.loc[100]
结果:
10000 loops, best of 3: 71.2 µs per loop
10 loops, best of 3: 38.9 ms per loop
10000 loops, best of 3: 134 µs per loop
这篇关于大 pandas 中非唯一索引的性能影响是什么?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!