如何并行合并两个 pandas 数据帧(多线程或多处理) [英] How to merge two pandas dataframe in parallel (multithreading or multiprocessing)

问题描述

如果不进行并行编程，我可以使用下面的代码合并 key 列上的左右数据框，但由于两者都非常大，所以速度会太慢.有什么办法可以有效地并行化吗?

Without doing in parallel programming I can merger left and right dataframe on key column using below code, but it will be too slow since both are very large. is there any way I can do it in parallelize efficiently ?

我有 64 个内核，所以实际上我可以使用其中的 63 个来合并这两个数据帧.

I have 64 cores, and so practically I can use 63 of them to merge these two dataframe.

left = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
                      'A': ['A0', 'A1', 'A2', 'A3'],
                     'B': ['B0', 'B1', 'B2', 'B3']})


right = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
                      'C': ['C0', 'C1', 'C2', 'C3'],
                      'D': ['D0', 'D1', 'D2', 'D3']})


result = pd.merge(left, right, on='key')

输出将是:

left:
    A   B key
0  A0  B0  K0
1  A1  B1  K1
2  A2  B2  K2
3  A3  B3  K3

right:
    C   D key
0  C0  D0  K0
1  C1  D1  K1
2  C2  D2  K2
3  C3  D3  K3

result:
    A   B key   C   D
0  A0  B0  K0  C0  D0
1  A1  B1  K1  C1  D1
2  A2  B2  K2  C2  D2
3  A3  B3  K3  C3  D3

我想并行执行此操作，以便快速完成.

I want to do this in parallel so I can do it at speed.

推荐答案

我相信你可以使用 dask.和函数 合并.

I believe you can use dask. and function merge.

文档说:

什么绝对有效?

巧妙的并行化操作(也很快):

Cleverly parallelizable operations (also fast):

加入索引:dd.merge(df1, df2, left_index=True, right_index=True)

或者:

需要 shuffle 的操作(慢，除非在索引上)

Operations requiring a shuffle (slow-ish, unless on index)

设置索引:df.set_index(df.x)

不加入索引:pd.merge(df1, df2, on='name')

您还可以查看如何创建 Dask 数据帧.

示例

import pandas as pd

left = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
                      'A': ['A0', 'A1', 'A2', 'A3'],
                     'B': ['B0', 'B1', 'B2', 'B3']})


right = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
                      'C': ['C0', 'C1', 'C2', 'C3'],
                      'D': ['D0', 'D1', 'D2', 'D3']})


result = pd.merge(left, right, on='key')
print result
    A   B key   C   D
0  A0  B0  K0  C0  D0
1  A1  B1  K1  C1  D1
2  A2  B2  K2  C2  D2
3  A3  B3  K3  C3  D3

import dask.dataframe as dd

#Construct a dask objects from a pandas objects
left1 = dd.from_pandas(left, npartitions=3)
right1 = dd.from_pandas(right, npartitions=3)

#merge on key
print dd.merge(left1, right1, on='key').compute()
    A   B key   C   D
0  A3  B3  K3  C3  D3
1  A1  B1  K1  C1  D1
0  A2  B2  K2  C2  D2
1  A0  B0  K0  C0  D0

#first set indexes and then merge by them
print dd.merge(left1.set_index('key').compute(), 
               right1.set_index('key').compute(), 
               left_index=True, 
               right_index=True)
      A   B   C   D
key                
K0   A0  B0  C0  D0
K1   A1  B1  C1  D1
K2   A2  B2  C2  D2
K3   A3  B3  C3  D3

这篇关于如何并行合并两个 pandas 数据帧(多线程或多处理)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！