数据帧上的 Apache Spark Python 余弦相似度 [英] Apache Spark Python Cosine Similarity over DataFrames

问题描述

对于推荐系统，我需要计算整个 Spark DataFrame 的所有列之间的余弦相似度.

For a Recommender System, I need to compute the cosine similarity between all the columns of a whole Spark DataFrame.

在 Pandas 中我曾经这样做过:

In Pandas I used to do this:

import sklearn.metrics as metrics
import pandas as pd

df= pd.DataFrame(...some dataframe over here :D ...)
metrics.pairwise.cosine_similarity(df.T,df.T)

这会生成列之间的相似度矩阵(因为我使用了转置)

That generates the Similarity Matrix between the columns (since I used the transposition)

有没有办法在 Spark (Python) 中做同样的事情?

Is there any way to do the same thing in Spark (Python)?

(我需要将此应用于由数千万行和数千列组成的矩阵，所以这就是我需要在 Spark 中执行此操作的原因)

(I need to apply this to a matrix made of tens of millions of rows, and thousands of columns, so that's why I need to do it in Spark)

推荐答案

您可以在 RowMatrix 上使用内置的 columnSimilarities() 方法，该方法既可以计算精确的余弦相似度，或使用 DIMSUM 方法估计它，这对于更大的数据集会快得多.用法的区别在于，对于后者，您必须指定一个 threshold.

You can use the built-in columnSimilarities() method on a RowMatrix, that can both calculate the exact cosine similarities, or estimate it using the DIMSUM method, which will be considerably faster for larger datasets. The difference in usage is that for the latter, you'll have to specify a threshold.

这是一个可重现的小例子:

Here's a small reproducible example:

from pyspark.mllib.linalg.distributed import RowMatrix
rows = sc.parallelize([(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, 11, 12)])

# Convert to RowMatrix
mat = RowMatrix(rows)

# Calculate exact and approximate similarities
exact = mat.columnSimilarities()
approx = mat.columnSimilarities(0.05)

# Output
exact.entries.collect()
[MatrixEntry(0, 2, 0.991935352214),
 MatrixEntry(1, 2, 0.998441152599),
 MatrixEntry(0, 1, 0.997463284056)]

这篇关于数据帧上的 Apache Spark Python 余弦相似度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！