将列表的列拆分为同一PySpark数据框中的多个列 [英] Split column of list into multiple columns in the same PySpark dataframe
本文介绍了将列表的列拆分为同一PySpark数据框中的多个列的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我有以下包含2列的数据框:
第一列具有列名
第二列具有值列表.
+--------------------+--------------------+
| Column| Quantile|
+--------------------+--------------------+
| rent|[4000.0, 4500.0, ...|
| is_rent_changed|[0.0, 0.0, 0.0, 0...|
| phone|[7.022372888E9, 7...|
| Area_house|[1000.0, 1000.0, ...|
| bedroom_count|[1.0, 1.0, 1.0, 1...|
| bathroom_count|[1.0, 1.0, 1.0, 1...|
| maintenance_cost|[0.0, 0.0, 0.0, 0...|
| latitude|[12.8217605, 12.8...|
| Max_rent|[9000.0, 10000.0,...|
| Beds|[2.0, 2.0, 2.0, 2...|
| Area|[1000.0, 1000.0, ...|
| Avg_Rent|[3500.0, 4000.0, ...|
| deposit_amount|[0.0, 0.0, 0.0, 0...|
| commission|[0.0, 0.0, 0.0, 0...|
| monthly_rent|[0.0, 0.0, 0.0, 0...|
|is_min_rent_guara...|[0.0, 0.0, 0.0, 0...|
|min_guarantee_amount|[0.0, 0.0, 0.0, 0...|
|min_guarantee_dur...|[1.0, 1.0, 1.0, 1...|
| furnish_cost|[0.0, 0.0, 0.0, 0...|
| owner_furnish_part|[0.0, 0.0, 0.0, 0...|
+--------------------+--------------------+
如何将第二列拆分为保留相同数据集的多列.
我可以使用:
univar_df10.select("Column", univar_df10.Quantile[0],univar_df10.Quantile[1],univar_df10.Quantile[2]).show()
+--------------------+-------------+-------------+------------+
| Column| Quantile[0]| Quantile[1]| Quantile[2]|
+--------------------+-------------+-------------+------------+
| rent| 4000.0| 4500.0| 5000.0|
| is_rent_changed| 0.0| 0.0| 0.0|
| phone|7.022372888E9|7.042022842E9|7.07333021E9|
| Area_house| 1000.0| 1000.0| 1000.0|
| bedroom_count| 1.0| 1.0| 1.0|
| bathroom_count| 1.0| 1.0| 1.0|
| maintenance_cost| 0.0| 0.0| 0.0|
| latitude| 12.8217605| 12.8490502| 12.863517|
| Max_rent| 9000.0| 10000.0| 11500.0|
| Beds| 2.0| 2.0| 2.0|
| Area| 1000.0| 1000.0| 1000.0|
| Avg_Rent| 3500.0| 4000.0| 4125.0|
| deposit_amount| 0.0| 0.0| 0.0|
| commission| 0.0| 0.0| 0.0|
| monthly_rent| 0.0| 0.0| 0.0|
|is_min_rent_guara...| 0.0| 0.0| 0.0|
|min_guarantee_amount| 0.0| 0.0| 0.0|
|min_guarantee_dur...| 1.0| 1.0| 1.0|
| furnish_cost| 0.0| 0.0| 0.0|
| owner_furnish_part| 0.0| 0.0| 0.0|
+--------------------+-------------+-------------+------------+
only showing top 20 rows
我希望我的新数据框将列表的第二列拆分为多个列，例如上述数据集. 预先感谢.
解决方案
假设(您的问题被标记为关闭，因为您不清楚您要问的是什么)，您的问题是您的Quantile列有一定长度，因此手动构建各个命令并不方便，这是一种使用列表加法和理解作为select的参数的解决方案:
spark.version
# u'2.2.1'
# make some toy data
from pyspark.sql import Row
df = spark.createDataFrame([Row([0,45,63,0,0,0,0]),
Row([0,0,0,85,0,69,0]),
Row([0,89,56,0,0,0,0])],
['features'])
df.show()
# result:
+-----------------------+
|features |
+-----------------------+
|[0, 45, 63, 0, 0, 0, 0]|
|[0, 0, 0, 85, 0, 69, 0]|
|[0, 89, 56, 0, 0, 0, 0]|
+-----------------------+
# get the length of your lists, if you don't know it already (here is 7):
length = len(df.select('features').take(1)[0][0])
length
# 7
df.select([df.features] + [df.features[i] for i in range(length)]).show()
# result:
+--------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| features|features[0]|features[1]|features[2]|features[3]|features[4]|features[5]|features[6]|
+--------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
|[0, 45, 63, 0, 0,...| 0| 45| 63| 0| 0| 0| 0|
|[0, 0, 0, 85, 0, ...| 0| 0| 0| 85| 0| 69| 0|
|[0, 89, 56, 0, 0,...| 0| 89| 56| 0| 0| 0| 0|
+--------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
所以，就您而言，
univar_df10.select([univar_df10.Column] + [univar_df10.Quantile[i] for i in range(length)])
在将length计算为
应该完成这项工作
length = len(univar_df10.select('Quantile').take(1)[0][0])
I have the following dataframe which contains 2 columns:
1st column has column names
2nd Column has list of values.
+--------------------+--------------------+
| Column| Quantile|
+--------------------+--------------------+
| rent|[4000.0, 4500.0, ...|
| is_rent_changed|[0.0, 0.0, 0.0, 0...|
| phone|[7.022372888E9, 7...|
| Area_house|[1000.0, 1000.0, ...|
| bedroom_count|[1.0, 1.0, 1.0, 1...|
| bathroom_count|[1.0, 1.0, 1.0, 1...|
| maintenance_cost|[0.0, 0.0, 0.0, 0...|
| latitude|[12.8217605, 12.8...|
| Max_rent|[9000.0, 10000.0,...|
| Beds|[2.0, 2.0, 2.0, 2...|
| Area|[1000.0, 1000.0, ...|
| Avg_Rent|[3500.0, 4000.0, ...|
| deposit_amount|[0.0, 0.0, 0.0, 0...|
| commission|[0.0, 0.0, 0.0, 0...|
| monthly_rent|[0.0, 0.0, 0.0, 0...|
|is_min_rent_guara...|[0.0, 0.0, 0.0, 0...|
|min_guarantee_amount|[0.0, 0.0, 0.0, 0...|
|min_guarantee_dur...|[1.0, 1.0, 1.0, 1...|
| furnish_cost|[0.0, 0.0, 0.0, 0...|
| owner_furnish_part|[0.0, 0.0, 0.0, 0...|
+--------------------+--------------------+
How do I split the second column into Multiple Columns Preserving the same dataset.
I can access the values using :
univar_df10.select("Column", univar_df10.Quantile[0],univar_df10.Quantile[1],univar_df10.Quantile[2]).show()
+--------------------+-------------+-------------+------------+
| Column| Quantile[0]| Quantile[1]| Quantile[2]|
+--------------------+-------------+-------------+------------+
| rent| 4000.0| 4500.0| 5000.0|
| is_rent_changed| 0.0| 0.0| 0.0|
| phone|7.022372888E9|7.042022842E9|7.07333021E9|
| Area_house| 1000.0| 1000.0| 1000.0|
| bedroom_count| 1.0| 1.0| 1.0|
| bathroom_count| 1.0| 1.0| 1.0|
| maintenance_cost| 0.0| 0.0| 0.0|
| latitude| 12.8217605| 12.8490502| 12.863517|
| Max_rent| 9000.0| 10000.0| 11500.0|
| Beds| 2.0| 2.0| 2.0|
| Area| 1000.0| 1000.0| 1000.0|
| Avg_Rent| 3500.0| 4000.0| 4125.0|
| deposit_amount| 0.0| 0.0| 0.0|
| commission| 0.0| 0.0| 0.0|
| monthly_rent| 0.0| 0.0| 0.0|
|is_min_rent_guara...| 0.0| 0.0| 0.0|
|min_guarantee_amount| 0.0| 0.0| 0.0|
|min_guarantee_dur...| 1.0| 1.0| 1.0|
| furnish_cost| 0.0| 0.0| 0.0|
| owner_furnish_part| 0.0| 0.0| 0.0|
+--------------------+-------------+-------------+------------+
only showing top 20 rows
I want my new dataframe to to split my 2nd column of lists into multiple columns like the above dataset. Thanks in advance.
解决方案
Assuming (your question is flagged for closure as unclear what you're asking) that your issue is that the lists in your Quantile column are of some length, and so it is not convenient to build the respective command by hand, here is a solution using list addition and comprehension as an argument to select:
spark.version
# u'2.2.1'
# make some toy data
from pyspark.sql import Row
df = spark.createDataFrame([Row([0,45,63,0,0,0,0]),
Row([0,0,0,85,0,69,0]),
Row([0,89,56,0,0,0,0])],
['features'])
df.show()
# result:
+-----------------------+
|features |
+-----------------------+
|[0, 45, 63, 0, 0, 0, 0]|
|[0, 0, 0, 85, 0, 69, 0]|
|[0, 89, 56, 0, 0, 0, 0]|
+-----------------------+
# get the length of your lists, if you don't know it already (here is 7):
length = len(df.select('features').take(1)[0][0])
length
# 7
df.select([df.features] + [df.features[i] for i in range(length)]).show()
# result:
+--------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
| features|features[0]|features[1]|features[2]|features[3]|features[4]|features[5]|features[6]|
+--------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
|[0, 45, 63, 0, 0,...| 0| 45| 63| 0| 0| 0| 0|
|[0, 0, 0, 85, 0, ...| 0| 0| 0| 85| 0| 69| 0|
|[0, 89, 56, 0, 0,...| 0| 89| 56| 0| 0| 0| 0|
+--------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+
So, in your case,
univar_df10.select([univar_df10.Column] + [univar_df10.Quantile[i] for i in range(length)])
should do the job, after you have calculated length as
length = len(univar_df10.select('Quantile').take(1)[0][0])
这篇关于将列表的列拆分为同一PySpark数据框中的多个列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
