如何在Spark DataFrame/DataSet中将行拆分为不同的列? [英] How to Split rows to different columns in Spark DataFrame/DataSet?

问题描述

假设我的数据集如下:

Name | Subject | Y1  | Y2 
A    | math    | 1998| 2000
B    |         | 1996| 1999
     | science | 2004| 2005

我想拆分此数据集的行，以便像这样消除Y2列:

I want to split rows of this data set such that Y2 column will be eliminated like :

Name | Subject | Y1
A    | math    | 1998
A    | math    | 1999
A    | math    | 2000
B    |         | 1996
B    |         | 1997
B    |         | 1998
B    |         | 1999
     | science | 2004
     | science | 2005

有人可以在这里提出一些建议吗?我希望我已经使我的查询清楚了.预先感谢.

Can someone suggest something here ? I hope I had made my query clear. Thanks in advance.

推荐答案

我认为您只需创建 udf 即可创建范围.然后，您可以使用explode创建必要的行:

I think you only need to create an udf to create the range. Then you can use explode to create the necessary rows:

val createRange = udf { (yearFrom: Int, yearTo: Int) =>
    (yearFrom to yearTo).toList
}

df.select($"Name", $"Subject", functions.explode(createRange($"Y1", $"Y2"))).show()

此代码的python版本类似于:

The python version of this code would be something like:

from pyspark.sql import Row
from pyspark.sql.functions import udf, explode
from pyspark.sql.types import IntegerType

createRange=udf( lambda (yearFrom, yearTo): list(range(yearFrom, yearTo)), IntegerType())

df.select($"Name", $"Subject", explode(createRange($"Y1", $"Y2"))).show()

这篇关于如何在Spark DataFrame/DataSet中将行拆分为不同的列?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！