在spark中将数组数据分解成行 [英] Explode array data into rows in spark

问题描述

我有以下方式的数据集:

FieldA FieldB ArrayField1 A {1,2,3}2 B {3,5}

我想分解 ArrayField 上的数据，以便输出如下所示:

FieldA FieldB ExplodedField1 一个 11 一个 21 一个 32 乙 32 乙 5

我的意思是我想为 ArrayField 中的数组中的每个项目生成一个输出行，同时保留其他字段的值.

您将如何在 Spark 中实现它.请注意，输入数据集非常大.

解决方案

explode 函数应该可以完成.

pyspark 版本:

<预><代码>>>>df = spark.createDataFrame([(1, "A", [1,2,3]), (2, "B", [3,5])],["col1", "col2", "col3"])>>>从 pyspark.sql.functions 导入爆炸>>>df.withColumn("col3",explode(df.col3)).show()+----+----+----+|col1|col2|col3|+----+----+----+|1|A|1||1|A|2||1|A|3||2|乙|3||2|乙|5|+----+----+----+

Scala 版本

scala>val df = Seq((1, "A", Seq(1,2,3)), (2, "B", Seq(3,5))).toDF("col1", "col2", "col3"")df: org.apache.spark.sql.DataFrame = [col1: int, col2: string ... 1 more field]标度>df.withColumn("col3",explode($"col3")).show()+----+----+----+|col1|col2|col3|+----+----+----+|1|A|1||1|A|2||1|A|3||2|乙|3||2|乙|5|+----+----+----+

I have a dataset in the following way:

FieldA    FieldB    ArrayField
1         A         {1,2,3}
2         B         {3,5}

I would like to explode the data on ArrayField so the output will look in the following way:

FieldA    FieldB    ExplodedField
1         A         1
1         A         2
1         A         3
2         B         3
2         B         5

I mean I want to generate an output line for each item in the array the in ArrayField while keeping the values of the other fields.

How would you implement it in Spark. Notice that the input dataset is very large.

解决方案

The explode function should get that done.

pyspark version:

>>> df = spark.createDataFrame([(1, "A", [1,2,3]), (2, "B", [3,5])],["col1", "col2", "col3"])
>>> from pyspark.sql.functions import explode
>>> df.withColumn("col3", explode(df.col3)).show()
+----+----+----+
|col1|col2|col3|
+----+----+----+
|   1|   A|   1|
|   1|   A|   2|
|   1|   A|   3|
|   2|   B|   3|
|   2|   B|   5|
+----+----+----+

Scala version

scala> val df = Seq((1, "A", Seq(1,2,3)), (2, "B", Seq(3,5))).toDF("col1", "col2", "col3")
df: org.apache.spark.sql.DataFrame = [col1: int, col2: string ... 1 more field]

scala> df.withColumn("col3", explode($"col3")).show()
+----+----+----+
|col1|col2|col3|
+----+----+----+
|   1|   A|   1|
|   1|   A|   2|
|   1|   A|   3|
|   2|   B|   3|
|   2|   B|   5|
+----+----+----+

这篇关于在spark中将数组数据分解成行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！