Spark中两个DStream的笛卡尔积 [英] Cartesian product of two DStream in Spark

问题描述

如何在Apache流中生成两个DStream，例如 cartesian(RDD< U>)，当在类型T和U的数据集上调用时，该数据流将返回(T，U)对的数据集(全部对元素).

How I can product two DStream in apache streaming like cartesian(RDD<U>) which when called on datasets of types T and U, returns a dataset of (T, U) pairs (all pairs of elements).

一种解决方案是使用join，但效果似乎不太好.

One solution is using join as follow that doesn't seem good.

    JavaPairDStream<Integer, String> xx = DStream_A.mapToPair(s -> {
        return new Tuple2<>(1, s);
    });

    JavaPairDStream<Integer, String> yy = DStream_B.mapToPair(e -> {
        return new Tuple2<>(1, e);
    });

    DStream_A_product_B = xx.join(yy);

有没有更好的解决方案?或如何使用RDD的笛卡尔方法?

Is there any better solution? or how i can use Cartesian method of RDD?

推荐答案

我找到了答案:

JavaPairDStream<String, String> cartes = DStream_A.transformWithToPair(DStream_B, 
     new Function3<JavaPairRDD<String, String>, JavaRDD<String>, Time, JavaPairRDD<String, String>>() {
        @Override
        public JavaPairRDD<String, String> call(JavaRDD<String> rddA, JavaRDD<String> rddB, Time v3) throws Exception {
            JavaPairRDD<String, String> res = rddA.cartesian(rddB);
            return res;
        }
    });

这篇关于Spark中两个DStream的笛卡尔积的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！