来自 List<String> 的数据框在 Java 中 [英] Dataframe from List&lt;String&gt; in Java

问题描述

• Spark 版本:1.6.2
• Java 版本:7

我有一个 List 数据.类似的东西:

I have a List<String> data. Something like:

[[dev, engg, 10000], [karthik, engg, 20000]..]

我知道这些数据的架构.

I know schema for this data.

name (String)
degree (String)
salary (Integer)

<小时>

我试过了:

---

I tried:

JavaRDD<String> data = new JavaSparkContext(sc).parallelize(datas);
DataFrame df = sqlContext.read().json(data);
df.printSchema();
df.show(false);

输出:

root
 |-- _corrupt_record: string (nullable = true)


+-----------------------------+
|_corrupt_record              |
+-----------------------------+
|[dev, engg, 10000]           |
|[karthik, engg, 20000]       |
+-----------------------------+

因为 List 不是正确的 JSON.

Because List<String> is not a proper JSON.

我需要创建一个合适的 JSON 还是有其他方法可以做到这一点?

推荐答案

您可以从 List 创建 DataFrame，然后使用 selectExpr 和 split 以获取所需的 DataFrame.

You can create DataFrame from List<String> and then use selectExpr and split to get desired DataFrame.

public class SparkSample{
public static void main(String[] args) {
    SparkConf conf = new SparkConf().setAppName("SparkSample").setMaster("local[*]");
    JavaSparkContext jsc = new JavaSparkContext(conf);
    SQLContext sqc = new SQLContext(jsc);
    // sample data
    List<String> data = new ArrayList<String>();
    data.add("dev, engg, 10000");
    data.add("karthik, engg, 20000");
    // DataFrame
    DataFrame df = sqc.createDataset(data, Encoders.STRING()).toDF();
    df.printSchema();
    df.show();
    // Convert
    DataFrame df1 = df.selectExpr("split(value, ',')[0] as name", "split(value, ',')[1] as degree","split(value, ',')[2] as salary");
    df1.printSchema();
    df1.show(); 
   }
}

您将得到以下输出.

root
 |-- value: string (nullable = true)

+--------------------+
|               value|
+--------------------+
|    dev, engg, 10000|
|karthik, engg, 20000|
+--------------------+

root
 |-- name: string (nullable = true)
 |-- degree: string (nullable = true)
 |-- salary: string (nullable = true)

+-------+------+------+
|   name|degree|salary|
+-------+------+------+
|    dev|  engg| 10000|
|karthik|  engg| 20000|
+-------+------+------+

您提供的示例数据有空格.如果您想删除空格并将工资类型设为整数"，那么您可以使用 trim 和 cast 函数，如下所示.

The sample data you have provided has empty spaces. If you want to remove space and have the salary type as "integer" then you can use trim and cast function like below.

df1 = df1.select(trim(col("name")).as("name"),trim(col("degree")).‌​as("degree"),trim(co‌​l("salary")).cast("i‌​nteger").as("salary"‌​)); 

这篇关于来自 List<String> 的数据框在 Java 中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！