PySpark-使用UDF从值列表中添加列 [英] PySpark - Adding a Column from a list of values using a UDF
本文介绍了PySpark-使用UDF从值列表中添加列的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我必须基于值列表将列添加到PySpark数据框.
I have to add column to a PySpark dataframe based on a list of values.
a= spark.createDataFrame([("Dog", "Cat"), ("Cat", "Dog"), ("Mouse", "Cat")],["Animal", "Enemy"])
我有一个称为评价的列表,它是每个宠物的评价.
I have a list called rating, which is a rating of each pet.
rating = [5,4,1]
我需要在数据框后面附加一个称为Rating的列,这样
I need to append the dataframe with a column called Rating, such that
+------+-----+------+
|Animal|Enemy|Rating|
+------+-----+------+
| Dog| Cat| 5|
| Cat| Dog| 4|
| Mouse| Cat| 1|
+------+-----+------+
我已经完成以下操作,但是它仅返回评级"列中列表中的第一个值
I have done the following however it is returning only the first value in the list in the Rating Column
def add_labels():
return rating.pop(0)
labels_udf = udf(add_labels, IntegerType())
new_df = a.withColumn('Rating', labels_udf()).cache()
退出:
+------+-----+------+
|Animal|Enemy|Rating|
+------+-----+------+
| Dog| Cat| 5|
| Cat| Dog| 5|
| Mouse| Cat| 5|
+------+-----+------+
推荐答案
希望这会有所帮助!
from pyspark.sql.functions import monotonically_increasing_id, row_number
#sample data
a= sqlContext.createDataFrame([("Dog", "Cat"), ("Cat", "Dog"), ("Mouse", "Cat")],
["Animal", "Enemy"])
a.show()
#convert list to a dataframe
rating = [5,4,1]
b = sqlContext.createDataFrame([(l,) for l in rating], ['Rating'])
#add 'sequential' index and join both dataframe to get the final result
a = a.withColumn("row_idx", row_number().over(Window.orderBy(monotonically_increasing_id())))
b = b.withColumn("row_idx", row_number().over(Window.orderBy(monotonically_increasing_id())))
final_df = a.join(b, a.row_idx == b.row_idx).\
drop("row_idx")
final_df.show()
输入:
+------+-----+
|Animal|Enemy|
+------+-----+
| Dog| Cat|
| Cat| Dog|
| Mouse| Cat|
+------+-----+
输出为:
+------+-----+------+
|Animal|Enemy|Rating|
+------+-----+------+
| Cat| Dog| 4|
| Dog| Cat| 5|
| Mouse| Cat| 1|
+------+-----+------+
这篇关于PySpark-使用UDF从值列表中添加列的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文