在 pandas 中有效地使用替换 [英] Using replace efficiently in pandas
问题描述
我希望在 python3 中以有效的方式使用 replace
函数.我拥有的代码正在完成任务,但速度太慢,因为我正在处理一个大型数据集.因此,在权衡时,我的首要任务是效率而不是优雅.这是我想做的一个玩具:
I am looking to use the replace
function in an efficient way in python3. The code I have is achieving the task, but is much too slow, as I am working with a large dataset. Thus, my priority is efficiency over elegancy whenever there is a tradeoff. Here is a toy of what I would like to do:
import pandas as pd
df = pd.DataFrame([[1,2],[3,4],[5,6]], columns = ['1st', '2nd'])
1st 2nd
0 1 2
1 3 4
2 5 6
idxDict= dict()
idxDict[1] = 'a'
idxDict[3] = 'b'
idxDict[5] = 'c'
for k,v in idxDict.items():
df ['1st'] = df ['1st'].replace(k, v)
哪个给了
1st 2nd
0 a 2
1 b 4
2 c 6
如我所愿,但需要的时间太长.最快的方法是什么?
as I desire, but it takes way too long. What would be the fastest way?
这是一个比这个更集中和干净的问题,对于解决方案类似.
this is a more focused and clean question than this one, for which the solution is similar.
推荐答案
use map
执行查找:
use map
to perform a lookup:
In [46]:
df['1st'] = df['1st'].map(idxDict)
df
Out[46]:
1st 2nd
0 a 2
1 b 4
2 c 6
为了避免没有有效密钥的情况,您可以传递 na_action='ignore'
to avoid the situation where there is no valid key you can pass na_action='ignore'
您也可以使用 df['1st'].replace(idxDict)
但要回答您关于效率的问题:
You can also use df['1st'].replace(idxDict)
but to answer you question about efficiency:
时间
In [69]:
%timeit df['1st'].replace(idxDict)
%timeit df['1st'].map(idxDict)
1000 loops, best of 3: 1.57 ms per loop
1000 loops, best of 3: 1.08 ms per loop
In [70]:
%%timeit
for k,v in idxDict.items():
df ['1st'] = df ['1st'].replace(k, v)
100 loops, best of 3: 3.25 ms per loop
所以在这里使用 map
比这里快 3 倍以上
So using map
is over 3x faster here
在更大的数据集上:
In [3]:
df = pd.concat([df]*10000, ignore_index=True)
df.shape
Out[3]:
(30000, 2)
In [4]:
%timeit df['1st'].replace(idxDict)
%timeit df['1st'].map(idxDict)
100 loops, best of 3: 18 ms per loop
100 loops, best of 3: 4.31 ms per loop
In [5]:
%%timeit
for k,v in idxDict.items():
df ['1st'] = df ['1st'].replace(k, v)
100 loops, best of 3: 18.2 ms per loop
对于 30K 行的 df,map
快约 4 倍,因此它比 replace
或循环更好地扩展
For 30K row df, map
is ~4x faster so it scales better than replace
or looping
这篇关于在 pandas 中有效地使用替换的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!