如何使用Java在Spark中合并或合并两个稀疏向量? [英] How to combine or merge two sparse vectors in Spark using Java?

问题描述

我使用了Java的API，即Apache-Spark 1.2.0，并创建了两个解析向量，如下所示.

I used the Java's API, i.e. Apache-Spark 1.2.0, and created two parse vectors as follows.

Vector v1 = Vectors.sparse(3, new int[]{0, 2}, new double[]{1.0, 3.0});
Vector v2 = Vectors.sparse(2, new int[]{0, 1}, new double[]{4,5});

如何获得通过组合v1和v2形成的新向量v3，因此结果应为:(5, [0,2,3,4],[1.0, 3.0, 4.0, 5.0])

How can I get a new vector v3 that is formed by combining v1 and v2, so the result should be: (5, [0,2,3,4],[1.0, 3.0, 4.0, 5.0])

推荐答案

我发现问题已经有一年了，并且仍然悬而未决.在这里，我通过自己编写一个辅助函数来解决此问题，如下所示.

I found the problem has been one year and is still pending. Here, I solved the problem by writing a helper function myself, as follows.

public static SparseVector combineSparseVectors(SparseVector... svs) {
    int size = 0;
    int nonzeros = 0;
    for (SparseVector sv : svs) {
        size += sv.size();
        nonzeros += sv.indices().length;
    }

    if (nonzeros != 0) {
        int[] indices = new int[nonzeros];
        double[] values = new double[nonzeros];

        int pointer_D = 0;
        int totalPt_D = 0;
        int pointer_V = 0;
        for (SparseVector sv : svs) {
            int[] indicesSV = sv.indices();
            for (int i : indicesSV) {
                indices[pointer_D++] = i + totalPt_D;
            }
            totalPt_D += sv.size();

            double[] valuesSV = sv.values();
            for (double d : valuesSV) {
                values[pointer_V++] = d;
            }

        }
        return new SparseVector(size, indices, values);
    } else {
        System.out.println("all zeroes");
        return null;
    }

}

这篇关于如何使用Java在Spark中合并或合并两个稀疏向量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！