如何加入 Pyspark 中的多个列? [英] How to join on multiple columns in Pyspark?

问题描述

我正在使用 Spark 1.3 并且想使用 python 接口 (SparkSQL) 加入多个列

I am using Spark 1.3 and would like to join on multiple columns using python interface (SparkSQL)

以下工作:

我首先将它们注册为临时表.

I first register them as temp tables.

numeric.registerTempTable("numeric")
Ref.registerTempTable("Ref")

test  = numeric.join(Ref, numeric.ID == Ref.ID, joinType='inner')

我现在想根据多个列加入它们.

I would now like to join them based on multiple columns.

我得到 SyntaxError: invalid syntax with this:

I get SyntaxError: invalid syntax with this:

test  = numeric.join(Ref,
   numeric.ID == Ref.ID AND numeric.TYPE == Ref.TYPE AND
   numeric.STATUS == Ref.STATUS ,  joinType='inner')

推荐答案

你应该使用 &/| 操作符并注意 运算符优先级(== 的优先级低于按位 AND 和 OR):

You should use & / | operators and be careful about operator precedence (== has lower precedence than bitwise AND and OR):

df1 = sqlContext.createDataFrame(
    [(1, "a", 2.0), (2, "b", 3.0), (3, "c", 3.0)],
    ("x1", "x2", "x3"))

df2 = sqlContext.createDataFrame(
    [(1, "f", -1.0), (2, "b", 0.0)], ("x1", "x2", "x3"))

df = df1.join(df2, (df1.x1 == df2.x1) & (df1.x2 == df2.x2))
df.show()

## +---+---+---+---+---+---+
## | x1| x2| x3| x1| x2| x3|
## +---+---+---+---+---+---+
## |  2|  b|3.0|  2|  b|0.0|
## +---+---+---+---+---+---+

这篇关于如何加入 Pyspark 中的多个列?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！