在Spark数据集中对数字字符串进行排序 [英] Sorting numeric String in Spark Dataset

问题描述

让我们假设我具有以下 Dataset :

Let's assume that I have the following Dataset:

+-----------+----------+
|productCode|    amount|
+-----------+----------+
|      XX-13|       300|
|       XX-1|       250|
|       XX-2|       410|
|       XX-9|        50|
|      XX-10|        35|
|     XX-100|       870|
+-----------+----------+

其中 productCode 是 String 类型，而 amount 是 Int .

如果要尝试按 productCode 进行排序，结果将是(并且由于 String 比较的性质，这是预期的结果):

If one will try to order this by productCode the result will be (and this is expected because of nature of String comparison):

def orderProducts(product: Dataset[Product]): Dataset[Product] = {
    product.orderBy("productCode")
}

// Output:
+-----------+----------+
|productCode|    amount|
+-----------+----------+
|       XX-1|       250|
|      XX-10|        35|
|     XX-100|       870|
|      XX-13|       300|
|       XX-2|       410|
|       XX-9|        50|
+-----------+----------+

如何考虑 Dataset API，如何按 productCode 的 Integer 部分排序的输出?

How can I get an output ordered by Integer part of the productCode like below considering Dataset API?

+-----------+----------+
|productCode|    amount|
+-----------+----------+
|       XX-1|       250|
|       XX-2|       410|
|       XX-9|        50|
|      XX-10|        35|
|      XX-13|       300|
|     XX-100|       870|
+-----------+----------+

推荐答案

在orderBy中使用表达式.检查一下:

Use the expression in the orderBy. Check this out:

scala> val df = Seq(("XX-13",300),("XX-1",250),("XX-2",410),("XX-9",50),("XX-10",35),("XX-100",870)).toDF("productCode", "amt")
df: org.apache.spark.sql.DataFrame = [productCode: string, amt: int]

scala> df.orderBy(split('productCode,"-")(1).cast("int")).show
+-----------+---+
|productCode|amt|
+-----------+---+
|       XX-1|250|
|       XX-2|410|
|       XX-9| 50|
|      XX-10| 35|
|      XX-13|300|
|     XX-100|870|
+-----------+---+


scala>

使用窗口功能，您可以做到

With window functions, you could do like

scala> df.withColumn("row1",row_number().over(Window.orderBy(split('productCode,"-")(1).cast("int")))).show(false)
18/12/10 09:25:07 WARN window.WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
+-----------+---+----+
|productCode|amt|row1|
+-----------+---+----+
|XX-1       |250|1   |
|XX-2       |410|2   |
|XX-9       |50 |3   |
|XX-10      |35 |4   |
|XX-13      |300|5   |
|XX-100     |870|6   |
+-----------+---+----+


scala>

请注意，spark抱怨将所有数据移至单个分区.

Note that spark complains of moving all data to single partition.

这篇关于在Spark数据集中对数字字符串进行排序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！