如何获得使用concat_ws生成的结果的大小? [英] How to get the size of result generated using concat_ws?

问题描述

我正在对COL1执行groupBy，并使用concat_ws获取COL2的串联列表.如何获取该列表中的值计数?这是我的代码:

I am performing groupBy on COL1 and getting the concatenated list of COL2 using concat_ws. How can I get the count of values in that list? Here's my code:

Dataset<Row> ds = df.groupBy("COL1").agg(org.apache.spark.sql.functions
    .concat_ws(",",org.apache.spark.sql.functions.collect_list("COL2")).as("sample"));

推荐答案

使用

大小(e:列):列返回数组或映射的长度.

size(e: Column): Column Returns length of array or map.

---

下面的示例在Scala中，您可以将其转换为Java，但是不管使用哪种编程语言，总体思路是完全相同的.

---

The following example is in Scala and am leaving it to you to convert it to Java, but the general idea is exactly the same regardless of the programming language.

val input = spark.range(4)
  .withColumn("COL1", $"id" % 2)
  .select($"COL1", $"id" as "COL2")
scala> input.show
+----+----+
|COL1|COL2|
+----+----+
|   0|   0|
|   1|   1|
|   0|   2|
|   1|   3|
+----+----+

val s = input
  .groupBy("COL1")
  .agg(
    concat_ws(",", collect_list("COL2")) as "concat",
    size(collect_list("COL2")) as "size") // <-- size
scala> s.show
+----+------+----+
|COL1|concat|size|
+----+------+----+
|   0|   0,2|   2|
|   1|   1,3|   2|
+----+------+----+

---

在Java中如下所示.感谢 Krishna Prasad 与SO/Spark社区共享代码！

---

In Java that'd be as follows. Thanks Krishna Prasad for sharing the code with the SO/Spark community!

Dataset<Row> ds = df.groupBy("COL1").agg(
  org.apache.spark.sql.functions.concat_ws(",",org.apache.spark.sql.functions.collect_list("‌​COL2")).as("sample")‌​, 
  org.apache.spark.sql.functions.size(org.apache.spark.sql.functions.collect_list("COL2‌​")).as("size"));

这篇关于如何获得使用concat_ws生成的结果的大小?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！