将CSV数据加载到Dataframe中，然后使用Apache Spark(Java)转换为Array [英] Load CSV data in to Dataframe and convert to Array using Apache Spark (Java)

问题描述

我有一个包含以下数据的CSV文件:

I have a CSV file with below data :

1,2,5  
2,4  
2,3 

我想将它们加载到具有数组字符串模式的数据框中

I want to load them into a Dataframe having schema of string of array

输出应如下所示.

[1, 2, 5]  
[2, 4]  
[2, 3] 

已在此处使用scala对此进行了回答: Spark:将字符串列转换为数组

This has been answered using scala here: Spark: Convert column of string to an array

我想让它在Java中实现.
请帮助

I want to make it happen in Java.
Please help

推荐答案

下面是Java中的示例代码.您需要使用spark.read().text(String path)方法读取文件，然后调用split函数.

Below is the sample code in Java. You need to read your file using spark.read().text(String path) method and then call the split function.

import static org.apache.spark.sql.functions.split;

public class SparkSample {
    public static void main(String[] args) {
        SparkSession spark = SparkSession
                .builder()
                .appName("SparkSample")
                .master("local[*]")
                .getOrCreate();
        //Read file
        Dataset<Row> ds = spark.read().text("c://tmp//sample.csv").toDF("value");
        ds.show(false);     
        Dataset<Row> ds1 = ds.select(split(ds.col("value"), ",")).toDF("new_value");
        ds1.show(false);
        ds1.printSchema();
    }
}

这篇关于将CSV数据加载到Dataframe中，然后使用Apache Spark(Java)转换为Array的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！