将分隔的数据从文本文件读取到不同的RDD中 [英] Reading delimited data from text file into different RDDs

问题描述

Apache Spark Java的新手我有一个以空格分隔的文本文件，如下所示:

Am new to Apache Spark java I have a text file delimited by space as below

3,45.25,23.45
5,22.15,19.35
4,33.24,12.45
2,15.67,21.22

此处各栏的意思是:

• 第一列:索引值
• 第二列:纬度值
• 第3列:经度值

Am试图将此数据解析为2或3个RDD(或成对的RDD).到目前为止，这是我的代码:

Am trying to parse this data into 2 or 3 RDDs (or pair RDDs). This is my code so far:

JavaRDD<String> data = sc.textFile("hdfs://data.txt");

JavaRDD<Double> data1 = data.flatMap(
  new FlatMapFunction<String, Double>() {
    public Iterable<Double> call(Double data) {
      return Arrays.asList(data.split(","));
    }
});

推荐答案

这样的事情(使用Java 8以获得更好的可读性)?

Something like this (use Java 8 for better readability)?

JavaRDD<String> data = sc.textFile("hdfs://data.txt");
JavaRDD<Tuple3<Integer, Float, Float>> parsedData = data.map((line) -> line.split(","))
        .map((line) -> new Tuple3<>(parseInt(line[0]), parseFloat(line[1]), parseFloat(line[2])))
        .cache(); // Cache parsed to avoid recomputation in subsequent .mapToPair calls

JavaPairRDD<Integer, Float> latitudeByIndex = parsedData.mapToPair((line) -> new Tuple2<>(line._1(), line._2()));
JavaPairRDD<Integer, Float> longitudeByIndex = parsedData.mapToPair((line) -> new Tuple2<>(line._1(), line._3()));
JavaPairRDD<Integer, Tuple2<Float, Float>> pointByIndex = parsedData.mapToPair((line) -> new Tuple2<>(line._1(), new Tuple2<>(line._2(), line._3())));

这篇关于将分隔的数据从文本文件读取到不同的RDD中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！