pyspark 提取 ROC 曲线? [英] pyspark extract ROC curve?

问题描述

有没有办法从 pyspark 中的 Spark ML 获取 ROC 曲线上的点?在文档中，我看到了 Scala 而不是 python 的示例:https://spark.apache.org/docs/2.1.0/mllib-evaluation-metrics.html

Is there a way to get the points on an ROC curve from Spark ML in pyspark? In the documentation I see an example for Scala but not python: https://spark.apache.org/docs/2.1.0/mllib-evaluation-metrics.html

是吗?我当然可以想出实现它的方法，但我不得不想象如果有一个预先构建的函数它会更快.我正在处理 300 万个分数和几十个模型，所以速度很重要.

Is that right? I can certainly think of ways to implement it but I have to imagine it’s faster if there’s a pre-built function. I’m working with 3 million scores and a few dozen models so speed matters.

推荐答案

只要 ROC 曲线是 FPR 对 TPR 的图，就可以提取所需的值如下:

As long as the ROC curve is a plot of FPR against TPR, you can extract the needed values as following:

your_model.summary.roc.select('FPR').collect()
your_model.summary.roc.select('TPR').collect())

其中 your_model 可以是例如你从这样的东西中得到的模型:

Where your_model could be for example a model you got from something like this:

from pyspark.ml.classification import LogisticRegression
log_reg = LogisticRegression()
your_model = log_reg.fit(df)

现在你应该只绘制 FPR 对 TPR，例如使用 matplotlib.

Now you should just plot FPR against TPR, using for example matplotlib.

附言

这是使用名为 your_model(以及其他任何东西！)的模型绘制 ROC 曲线的完整示例.我还在 ROC 图中绘制了参考随机猜测"线.

Here is a complete example for plotting ROC curve using a model named your_model (and anything else!). I've also plot a reference "random guess" line inside the ROC plot.

import matplotlib.pyplot as plt
plt.figure(figsize=(5,5))
plt.plot([0, 1], [0, 1], 'r--')
plt.plot(your_model.summary.roc.select('FPR').collect(),
         your_model.summary.roc.select('TPR').collect())
plt.xlabel('FPR')
plt.ylabel('TPR')
plt.show()

这篇关于pyspark 提取 ROC 曲线?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！