如何在Spark Scala中将行数据转置/透视到列? [英] How to transpose/pivot the rows data to column in Spark Scala?
本文介绍了如何在Spark Scala中将行数据转置/透视到列?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我是Spark-SQL的新手.我在Spark Dataframe中有这样的信息
Company Type Status
A X done
A Y done
A Z done
C X done
C Y done
B Y done
我想像下面这样显示
Company X-type Y-type Z-type
A done done done
B pending done pending
C done done pending
我无法实现这是Spark-SQL
请帮助
解决方案
您可以groupby
公司,然后在类型
这是简单的示例
import org.apache.spark.sql.functions._
val df = spark.sparkContext.parallelize(Seq(
("A", "X", "done"),
("A", "Y", "done"),
("A", "Z", "done"),
("C", "X", "done"),
("C", "Y", "done"),
("B", "Y", "done")
)).toDF("Company", "Type", "Status")
val result = df.groupBy("Company")
.pivot("Type")
.agg(expr("coalesce(first(Status), \"pending\")"))
result.show()
输出:
+-------+-------+----+-------+
|Company| X| Y| Z|
+-------+-------+----+-------+
| B|pending|done|pending|
| C| done|done|pending|
| A| done|done| done|
+-------+-------+----+-------+
您以后可以重命名该列.
希望这会有所帮助!
I am new to Spark-SQL. I have information in Spark Dataframe like this
Company Type Status
A X done
A Y done
A Z done
C X done
C Y done
B Y done
I am want to be displayed like the following
Company X-type Y-type Z-type
A done done done
B pending done pending
C done done pending
I am not able to acheive this is Spark-SQL
Please Help
解决方案
You can groupby
Company and then use pivot
function on column Type
Here is the simple example
import org.apache.spark.sql.functions._
val df = spark.sparkContext.parallelize(Seq(
("A", "X", "done"),
("A", "Y", "done"),
("A", "Z", "done"),
("C", "X", "done"),
("C", "Y", "done"),
("B", "Y", "done")
)).toDF("Company", "Type", "Status")
val result = df.groupBy("Company")
.pivot("Type")
.agg(expr("coalesce(first(Status), \"pending\")"))
result.show()
Output:
+-------+-------+----+-------+
|Company| X| Y| Z|
+-------+-------+----+-------+
| B|pending|done|pending|
| C| done|done|pending|
| A| done|done| done|
+-------+-------+----+-------+
You can rename the column later.
Hope this helps!
这篇关于如何在Spark Scala中将行数据转置/透视到列?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文