将数组数据分解为Spark中的行 [英] Explode array data into rows in spark

问题描述

我通过以下方式获得数据集:

I have a dataset in the following way:

FieldA    FieldB    ArrayField
1         A         {1,2,3}
2         B         {3,5}

我想爆炸 ArrayField 上的数据，因此输出将以以下方式显示:

I would like to explode the data on ArrayField so the output will look in the following way:

FieldA    FieldB    ExplodedField
1         A         1
1         A         2
1         A         3
2         B         3
2         B         5

我的意思是我想为 ArrayField 中的数组中的每个项目生成一条输出行，同时保留其他字段的值.

I mean I want to generate an output line for each item in the array the in ArrayField while keeping the values of the other fields.

您将如何在Spark中实施它. 请注意，输入数据集非常大.

How would you implement it in Spark. Notice that the input dataset is very large.

推荐答案

pyspark版本:

>>> df = spark.createDataFrame([(1, "A", [1,2,3]), (2, "B", [3,5])],["col1", "col2", "col3"])
>>> from pyspark.sql.functions import explode
>>> df.withColumn("col3", explode(df.col3)).show()
+----+----+----+
|col1|col2|col3|
+----+----+----+
|   1|   A|   1|
|   1|   A|   2|
|   1|   A|   3|
|   2|   B|   3|
|   2|   B|   5|
+----+----+----+

Scala版本

scala> val df = Seq((1, "A", Seq(1,2,3)), (2, "B", Seq(3,5))).toDF("col1", "col2", "col3")
df: org.apache.spark.sql.DataFrame = [col1: int, col2: string ... 1 more field]

scala> df.withColumn("col3", explode($"col3")).show()
+----+----+----+
|col1|col2|col3|
+----+----+----+
|   1|   A|   1|
|   1|   A|   2|
|   1|   A|   3|
|   2|   B|   3|
|   2|   B|   5|
+----+----+----+

这篇关于将数组数据分解为Spark中的行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！