pandas 数据框:loc 与查询性能 [英] pandas dataframe: loc vs query performance

问题描述

我在 python 中有 2 个数据框，我想查询数据.

I have 2 dataframes in python that I would like to query for data.

• DF1:4M 记录 x 3 列.查询功能接缝更多比loc函数更高效.

• DF1: 4M records x 3 columns. The query function seams more efficient than the loc function.

DF2:2K 记录 x 6 列.loc 函数接缝更多比查询功能更高效.

DF2: 2K records x 6 columns. The loc function seams much more efficient than the query function.

两个查询都返回一条记录.模拟是通过在循环中运行相同的操作 10K 次来完成的.

Both queries return a single record. The simulation was done by running the same operation in a loop 10K times.

运行 python 2.7 和 Pandas 0.16.0

Running python 2.7 and pandas 0.16.0

有什么提高查询速度的建议吗?

Any recommendations to improve the query speed?

推荐答案

为了提高性能可以使用 numexpr:

For improve performance is possible use numexpr:

import numexpr

np.random.seed(125)
N = 40000000
df = pd.DataFrame({'A':np.random.randint(10, size=N)})

def ne(df):
    x = df.A.values
    return df[numexpr.evaluate('(x > 5)')]
print (ne(df))

In [138]: %timeit (ne(df))
1 loop, best of 3: 494 ms per loop

In [139]: %timeit df[df.A > 5]
1 loop, best of 3: 536 ms per loop

In [140]: %timeit df.query('A > 5')
1 loop, best of 3: 781 ms per loop

In [141]: %timeit df[df.eval('A > 5')]
1 loop, best of 3: 770 ms per loop

---

import numexpr
np.random.seed(125)

def ne(x):
    x = x.A.values
    return x[numexpr.evaluate('(x > 5)')]

def be(x):
    return x[x.A > 5]

def q(x):
    return x.query('A > 5')

def ev(x):
    return x[x.eval('A > 5')]


def make_df(n):
    df = pd.DataFrame(np.random.randint(10, size=n), columns=['A'])
    return df


perfplot.show(
    setup=make_df,
    kernels=[ne, be, q, ev],
    n_range=[2**k for k in range(2, 25)],
    logx=True,
    logy=True,
    equality_check=False,  
    xlabel='len(df)')

具有修改的ne和改进的be的图:

Graph with modified ne and improved be:

def ne(x):
    return x[numexpr.evaluate('(x > 5)')]

def bex(x):
    return x[x.A.values > 5]

这篇关于pandas 数据框:loc 与查询性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！