List< String>中的数据帧在Java中 [英] Dataframe from List<String> in Java

问题描述

• 火花版本:1.6.2
• Java版本:7

我有一个List<String>数据.像这样:

I have a List<String> data. Something like:

[[dev, engg, 10000], [karthik, engg, 20000]..]

我知道此数据的架构.

name (String)
degree (String)
salary (Integer)

---

我尝试过:

---

I tried:

JavaRDD<String> data = new JavaSparkContext(sc).parallelize(datas);
DataFrame df = sqlContext.read().json(data);
df.printSchema();
df.show(false);

输出:

root
 |-- _corrupt_record: string (nullable = true)


+-----------------------------+
|_corrupt_record              |
+-----------------------------+
|[dev, engg, 10000]           |
|[karthik, engg, 20000]       |
+-----------------------------+

因为List<String>不是正确的JSON.

Because List<String> is not a proper JSON.

我需要创建一个正确的JSON还是有其他方法可以做到这一点?

推荐答案

您可以从List<String>创建DataFrame，然后使用selectExpr和split获得所需的DataFrame.

You can create DataFrame from List<String> and then use selectExpr and split to get desired DataFrame.

public class SparkSample{
public static void main(String[] args) {
    SparkConf conf = new SparkConf().setAppName("SparkSample").setMaster("local[*]");
    JavaSparkContext jsc = new JavaSparkContext(conf);
    SQLContext sqc = new SQLContext(jsc);
    // sample data
    List<String> data = new ArrayList<String>();
    data.add("dev, engg, 10000");
    data.add("karthik, engg, 20000");
    // DataFrame
    DataFrame df = sqc.createDataset(data, Encoders.STRING()).toDF();
    df.printSchema();
    df.show();
    // Convert
    DataFrame df1 = df.selectExpr("split(value, ',')[0] as name", "split(value, ',')[1] as degree","split(value, ',')[2] as salary");
    df1.printSchema();
    df1.show(); 
   }
}

您将获得以下输出.

root
 |-- value: string (nullable = true)

+--------------------+
|               value|
+--------------------+
|    dev, engg, 10000|
|karthik, engg, 20000|
+--------------------+

root
 |-- name: string (nullable = true)
 |-- degree: string (nullable = true)
 |-- salary: string (nullable = true)

+-------+------+------+
|   name|degree|salary|
+-------+------+------+
|    dev|  engg| 10000|
|karthik|  engg| 20000|
+-------+------+------+

您提供的样本数据有空格.如果要删除空间并将薪金类型设置为整数"，则可以使用如下所示的trim和cast函数.

The sample data you have provided has empty spaces. If you want to remove space and have the salary type as "integer" then you can use trim and cast function like below.

df1 = df1.select(trim(col("name")).as("name"),trim(col("degree")).‌​as("degree"),trim(co‌​l("salary")).cast("i‌​nteger").as("salary"‌​)); 

这篇关于List< String>中的数据帧在Java中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！