将多行合并为单行 [英] Combine multiple rows into a single row

问题描述

我正在尝试通过pyspark构建sql实现此目的.目标是将多行合并为单行例子:我要转换

I am trying to achieve this via pyspark building sql. The goal is to combine multiple rows into single row Example: I want to convert this

+-----+----+----+-----+
| col1|col2|col3| col4|
+-----+----+----+-----+
|x    |  y |  z |13::1|
|x    |  y |  z |10::2|
+-----+----+----+-----+

收件人

+-----+----+----+-----------+
| col1|col2|col3|       col4|
+-----+----+----+-----------+
|x    |  y |  z |13::1;10::2|
+-----+----+----+-----------+

推荐答案

您正在寻找的是的spark-sql版本这个答案，如下:

What you're looking for is the spark-sql version of this answer, which is the following:

query = """
  select col1, 
         col2, 
         col3, 
         concat_ws(';', collect_list(col4)) as col4 
    from some_table 
group by col1, 
         col2, 
         col3
"""
spark.sql(query).show()
#+----+----+----+-----------+
#|col1|col2|col3|       col4|
#+----+----+----+-----------+
#|   x|   y|   z|13::1;10::2|
#+----+----+----+-----------+

但是请注意，由于分配了spark，因此除非您明确指定顺序，否则不能保证保持任何特定顺序.

But be aware that since spark is distributed, this is not guaranteed to maintain any specific order, unless you explicitly specify the order.

查看更多:

• 通过保留基于另一个变量的订单来收集列表
• collect_list()是否保持行的相对顺序?
• Spark DataFrame:groupBy在orderBy之后是否保持该顺序?

这篇关于将多行合并为单行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！