如何获取使用 concat_ws 生成的结果的大小? [英] How to get the size of result generated using concat_ws?

问题描述

我正在 COL1 上执行 groupBy 并使用 concat_ws 获取 COL2 的串联列表.如何获得该列表中的值计数?这是我的代码:

I am performing groupBy on COL1 and getting the concatenated list of COL2 using concat_ws. How can I get the count of values in that list? Here's my code:

Dataset<Row> ds = df.groupBy("COL1").agg(org.apache.spark.sql.functions
    .concat_ws(",",org.apache.spark.sql.functions.collect_list("COL2")).as("sample"));

推荐答案

使用 size 函数.

size(e: Column): Column 返回数组或映射的长度.

size(e: Column): Column Returns length of array or map.

<小时>

下面的例子是在 Scala 中，我把它留给你把它转换成 Java，但不管编程语言如何，总体思路是完全一样的.

---

The following example is in Scala and am leaving it to you to convert it to Java, but the general idea is exactly the same regardless of the programming language.

val input = spark.range(4)
  .withColumn("COL1", $"id" % 2)
  .select($"COL1", $"id" as "COL2")
scala> input.show
+----+----+
|COL1|COL2|
+----+----+
|   0|   0|
|   1|   1|
|   0|   2|
|   1|   3|
+----+----+

val s = input
  .groupBy("COL1")
  .agg(
    concat_ws(",", collect_list("COL2")) as "concat",
    size(collect_list("COL2")) as "size") // <-- size
scala> s.show
+----+------+----+
|COL1|concat|size|
+----+------+----+
|   0|   0,2|   2|
|   1|   1,3|   2|
+----+------+----+

<小时>

在 Java 中，如下所示.感谢 Krishna Prasad 与 SO/Spark 社区分享代码！

---

In Java that'd be as follows. Thanks Krishna Prasad for sharing the code with the SO/Spark community!

Dataset<Row> ds = df.groupBy("COL1").agg(
  org.apache.spark.sql.functions.concat_ws(",",org.apache.spark.sql.functions.collect_list("‌​COL2")).as("sample")‌​, 
  org.apache.spark.sql.functions.size(org.apache.spark.sql.functions.collect_list("COL2‌​")).as("size"));

这篇关于如何获取使用 concat_ws 生成的结果的大小?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！