将JavaPairRDD转换为JavaRDD [英] Convert JavaPairRDD to JavaRDD

问题描述

我使用ElasticSearch-Hadoop Library从ElsticSearch获取数据。

I am fetching data from ElsticSearch using ElasticSearch-Hadoop Library.

JavaPairRDD<String, Map<String, Object>> esRDD = JavaEsSpark.esRDD(sc);

现在我有了JavaPairRDD。我想在这个RDD上使用来自MLLib的随机森林。
所以我将它转换为JavaPairRDD.toRDD（esRDD）这将给我RDD。
使用RDD我再次转换为JavaRDD

Now I have JavaPairRDD. I want to use Random Forest from MLLib on this RDD. So I am converting it to JavaPairRDD.toRDD(esRDD) this will give me RDD. Using RDD I am converting again to JavaRDD

JavaRDD<LabeledPoint>[] splits = (JavaRDD.fromRDD(JavaPairRDD.toRDD(esRDD),
            esRDD.classTag())).randomSplit(new double[] { 0.5, 0.5 });

JavaRDD<LabeledPoint> trainingData = splits[0];
JavaRDD<LabeledPoint> testData = splits[1];

我想将trainingData和TestData传递给Random Forest算法，但它在编译时给出了转换异常。 / p>

I want to pass trainingData and TestData to Random Forest algorithm but it gives casting exception at compile time.

类型不匹配：无法转换为
JavaRDD [Tuple2 [String，Map [String，Object] ]] []到
JavaRDD [LabeledPoint] []

Type mismatch: cannot convert from JavaRDD[Tuple2[String,Map[String,Object]]][] to JavaRDD[LabeledPoint][]

添加方括号小于和大于标志不起作用

Added square brackets as less than and greater than signs are not working

任何人都可以建议我采用正确的方式进行施法。我是Spark Datastrucutres的新手。

Could any one suggest me the proper way for Casting. I am new to Spark Datastrucutres.

推荐答案

您在JavaPairRDD列中有哪些数据？ JavaPairRDD是第一列和第二列之间的键/值映射，与普通RDD不同。

What data do you have in the JavaPairRDD columns? A JavaPairRDD is a key/value mapping between the first and second column, unlike a normal RDD.

您可能希望从JavaPairRDD中删除第一列，仅返回只有值列的JavaRDD。

You possibly want to drop off the first column from the JavaPairRDD, returning just JavaRDD with just the value column.

为此，只需执行以下操作：

To to this, simply run something like:

JavaRDD newRDD = esRDD .map（x => x._2）;

JavaRDD newRDD = esRDD.map(x => x._2);

或相当于创建没有第一列的新JavaRDD。

or equivalent to create a new JavaRDD without the first column.

这篇关于将JavaPairRDD转换为JavaRDD的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！