分隔数据星火排序 [英] Spark sorting of delimited data

问题描述

我是新来的火花。你可以给任何想法是低于code中的问题：

I am new to Spark. Can you give any idea what is the problem with below code:

val rawData="""USA | E001 | ABC DE | 19850607 | IT | $100
UK | E005 | CHAN CL | 19870512 | OP | $200
USA | E003 | XYZ AB | 19890101 | IT | $250
USA | E002 | XYZ AB | 19890705 | IT | $200"""
val sc = ...     
val data= rawData.split("\n")
val rdd= sc.parallelize(data)
val data1=rdd.flatMap(line=> line.split(" | "))
val data2 = data1.map(arr => (arr(2), arr.mkString(""))).sortByKey(false)
data2.saveAsTextFile("./sample_data1_output")

在这里， .sortByKey（假）不能正常工作和编译器给我的错误：

Here, .sortByKey(false) is not working and compiler gives me error:

[error] /home/admin/scala/spark-poc/src/main/scala/SparkApp.scala:26: value sortByKey is not a member of org.apache.spark.rdd.RDD[(String, String)]
[error] val data2 = data1.map(arr => (arr(2), arr.mkString(""))).sortByKey(false) 

问题是如何获得MappedRDD？或者什么对象我应该叫sortByKey（）？

Question is how to get MappedRDD? Or on what object should I call sortByKey()?

推荐答案

星火提供了更多的操作，如sortByKey（），关于对RDDS。这些操作都可以通过类调用PairRDDFunctions和星火采用隐式转换为自动执行RDD - > PairRDDFunctions包装

Spark provides additional operations, like sortByKey(), on RDDs of pairs. These operations are available through a class called PairRDDFunctions and Spark uses implicit conversions to automatically perform the RDD -> PairRDDFunctions wrapping.

要导入的隐式转换，以下行添加到您的程序顶部：

To import the implicit conversions, add the following lines to the top of your program:

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._

这是在Spark节目指南的部分上的键 - 值对工作的。

This is discussed in the Spark programming guide's section on Working with key-value pairs.

这篇关于分隔数据星火排序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！