“已解决的属性缺失"在 pySpark 上执行 join 时 [英] "resolved attribute(s) missing" when performing join on pySpark
问题描述
我有以下两个 pySpark 数据框:
I have the following two pySpark dataframe:
> df_lag_pre.columns
['date','sku','name','country','ccy_code','quantity','usd_price','usd_lag','lag_quantity']
> df_unmatched.columns
['alt_sku', 'alt_lag_quantity', 'country', 'ccy_code', 'name', 'usd_price']
现在我想在公共列上加入它们,所以我尝试以下操作:
Now I want to join them on common columns, so I try the following:
> df_lag_pre.join(df_unmatched, on=['name','country','ccy_code','usd_price'])
我收到以下错误消息:
AnalysisException: u'resolved attribute(s) price#3424 missing from country#3443,month#801,price#808,category#803,subcategory#804,page#805,date#280,link#809,name#806,quantity#807,ccy_code#3439,sku#3004,day#802 in operator !EvaluatePython PythonUDF#<lambda>(ccy_code#3439,price#3424), pythonUDF#811: string;'
显示此错误的一些列(例如价格)是另一个数据帧的一部分,df_lag
是从该数据帧构建的.我找不到有关如何解释此消息的任何信息,因此我们将不胜感激.
Some of the columns that show up on this error, such as price, were part of another dataframe from which df_lag
was built from. I can't find any info on how to interpret this message, so any help would be greatly appreciated.
推荐答案
pyspark中可以这样执行join,看看对你有没有用:
You can perform join this way in pyspark, Please see if this is useful for you:
df_lag_pre.alias("df1")
df_unmatched.alias("df2")
join_both = df1.join(df2, (col("df1.name") == col("df2.name")) & (col("df1.country") == col("df2.country")) & (col("df1.ccy_code") == col("df2.ccy_code")) & (col("df1.usd_price") == col("df2.usd_price")), 'inner')
更新:如果您收到 col not defined 错误,请使用下面的导入
Update: If you are getting col not defined error, please use below import
from pyspark.sql.functions import col
这篇关于“已解决的属性缺失"在 pySpark 上执行 join 时的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!