如何从 Spark Java 中的结构检索值? [英] How to retrieve a value from struct in Spark Java?

问题描述

我的数据集 ds 具有以下架构:

My Dataset ds has the following schema:

root
 |-- id: string (nullable = true)
 |-- type: string (nullable = true)
 |-- item: struct (nullable = true)
 |    |-- item: string (nullable = true)

示例:

{"id":"1","type": "aaa", "item": {"item":"11"}}
{"id":"2","type": "bbb", "item" : {"item":"12"}}

如何从结构中检索 item 以获得此结果?

How can I retrieve item from struct to get this result?

id   type    item
1    aaa     11
2    bbb     12

这是我尝试但没有成功的方法:

This is what I tried without success:

ds.select("id", "type", "item.0");

请注意，我使用的是 Java.不要在 Scala 或 Python 中发布答案，除非它们在 Java 中是相同的.

Please notice that I use Java. Do not post answers in Scala or Python unless they are identical for Java.

推荐答案

假设您有示例文件:

{"id":"1","type": "aaa", "item": {"item":"11"}}
{"id":"2","type": "bbb", "item" : {"item":"12"}}

您可以测试以下 Java 代码:

You can test the following Java code:

public class SparkJavaTest {
    public static SparkSession spark = SparkSession
            .builder()
            .appName("JavaSparkTest")
            .master("local")
            .getOrCreate();


    public static void main(String[] args) {

    Dataset<Row> ds1 = spark.read().json("sample.json");

ds1.printSchema();

ds1.select("id", "type", "item.item").show(false);

结果是:

root
 |-- id: string (nullable = true)
 |-- item: struct (nullable = true)
 |    |-- item: string (nullable = true)
 |-- type: string (nullable = true)


+---+----+----+
|id |type|item|
+---+----+----+
|1  |aaa |11  |
|2  |bbb |12  |
+---+----+----+

这篇关于如何从 Spark Java 中的结构检索值?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！