将多行合并为一行 [英] Combine multiple rows into a single row

问题描述

我正在尝试通过 pyspark 构建 sql 来实现这一点.目标是将多行合并成单行例子:我想转换这个

I am trying to achieve this via pyspark building sql. The goal is to combine multiple rows into single row Example: I want to convert this

+-----+----+----+-----+
| col1|col2|col3| col4|
+-----+----+----+-----+
|x    |  y |  z |13::1|
|x    |  y |  z |10::2|
+-----+----+----+-----+

到

+-----+----+----+-----------+
| col1|col2|col3|       col4|
+-----+----+----+-----------+
|x    |  y |  z |13::1;10::2|
+-----+----+----+-----------+

推荐答案

您正在寻找的是 spark-sql 版本这个答案，如下:

What you're looking for is the spark-sql version of this answer, which is the following:

query = """
  select col1, 
         col2, 
         col3, 
         concat_ws(';', collect_list(col4)) as col4 
    from some_table 
group by col1, 
         col2, 
         col3
"""
spark.sql(query).show()
#+----+----+----+-----------+
#|col1|col2|col3|       col4|
#+----+----+----+-----------+
#|   x|   y|   z|13::1;10::2|
#+----+----+----+-----------+

但请注意，由于 spark 是分布式的，因此不能保证保持任何特定顺序，除非您明确指定顺序.

But be aware that since spark is distributed, this is not guaranteed to maintain any specific order, unless you explicitly specify the order.

查看更多:

• collect_list 根据另一个变量保留顺序
• collect_list() 是否维护行的相对顺序?
• Spark DataFrame:orderBy 之后的 groupBy 是否保持该顺序?

这篇关于将多行合并为一行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！