类型不匹配的降低 [英] Type Mismatch for Reduce

问题描述

我有一个RDD：

JavaRDD<Tuple2<Tuple2<String, Long>, Long>> mappedRdd = dataRDD
    .values().map(mapFunc);

我要在其上运行一个减少功能：

I want to run a reduce function on it:

private static Function2<Tuple2<Tuple2<String, Long>, Long>, Tuple2<Tuple2<String, Long>, Long>, Tuple2<Tuple2<String, Long>, Long>> redFunc2 = new Function2<Tuple2<Tuple2<String, Long>, Long>, Tuple2<Tuple2<String, Long>, Long>, Tuple2<Tuple2<String, Long>, Long>>() {

@Override
public Tuple2<String, MetricDatum> call(
  Tuple2<Tuple2<String, Long>, Long> v1,
  Tuple2<Tuple2<String, Long>, Long> v2) throws Exception {
  long sum = 0L; // sum up the values
  sum += v1._2();
  sum += v2._2();

  String dimension = v1._1()._1();
  long timestamp = v1._1()._2();

  MetricDatum metricDatum = new MetricDatum();
  metricDatum.setMetricDimension(dimension);
  metricDatum.setTimestamp(timestamp);

  String key = metricDatum.getMetricDimension().toString();
  key += "_" + Long.toString(timestamp);
  metricDatum.setMetric(sum);
  return new Tuple2<>(key, metricDatum);
}

};

但是它给出错误：

However it gives error:

JavaRDD<Tuple2<Tuple2<String, Long>, Long>>  reducedGoraRdd = mappedRdd.reduce(redFunc);

我想要做的火花这个例子<一个href=\"https://github.com/apache/gora/blob/master/gora-tutorial/src/main/java/org/apache/gora/tutorial/log/LogAnalytics.java\"相对=nofollow> LogAnalytics.java

我做错过什么，我应该使用flatMap等，或者减少功能是完全错误的？

Do I miss anything, should I use flatMap etc. or that reduce function is totally wrong?

推荐答案

从 LogAnalytics.java 减少函数库我写了某事是这样的：

Base on reduce function from LogAnalytics.java I wrote sth like this:

//dummy 
class MetricDatum {
    public void setMetricDimension(String l) {}
    public void setTimestamp(Long l) {}
    public void setMetric(Long l) {}
    public Object getMetricDimension() {return new Object();}
}
//fake input
JavaRDD<Tuple2<Tuple2<String, Long>, Long>> mappedRdd = sc.emptyRDD();

//creating JavaPairRDD from JavaRDD of pairs
JavaPairRDD.fromJavaRDD(mappedRdd)
//reduce with commutative, associative function (Long, Long) -> Long
.reduceByKey(new Function2<Long, Long, Long>() {
    @Override
    public Long call(Long aLong, Long aLong2) throws Exception {
        return aLong + aLong2;
    }
})
//map (key, sum) pairs to (newKey, metricDatum(sum)) and creatring JavaPairRDD
.mapToPair(new PairFunction<Tuple2<Tuple2<String,Long>,Long>, String, MetricDatum>() {
    @Override
    public Tuple2<String, MetricDatum> 
            call(Tuple2<Tuple2<String, Long>, Long> tuple2LongTuple2) throws Exception {
        String dimension = tuple2LongTuple2._1()._1();
        long timestamp = tuple2LongTuple2._1()._2();

        MetricDatum metricDatum = new MetricDatum();
        metricDatum.setMetricDimension(dimension);
        metricDatum.setTimestamp(timestamp);

        String key = metricDatum.getMetricDimension().toString();
        key += "_" + Long.toString(timestamp);
        metricDatum.setMetric(tuple2LongTuple2._2());
        return new Tuple2<String, MetricDatum>(key, metricDatum);
    }
});

这篇关于类型不匹配的降低的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！