如何在 spark.mllib 中的分类模型中获得每个实例的概率 [英] How to get the probability per instance in classifications models in spark.mllib

问题描述

我使用 spark.mllib.classification.{LogisticRegressionModel, LogisticRegressionWithSGD} 和 spark.mllib.tree.RandomForest 进行分类.使用这些包，我生成了分类模型.只有这些模型可以预测每个实例的特定类.在 Weka 中，我们可以获得每个实例属于每个类的准确概率.我们如何使用这些包来做到这一点?

I'm using spark.mllib.classification.{LogisticRegressionModel, LogisticRegressionWithSGD} and spark.mllib.tree.RandomForest for classification. Using these packages I produce classification models. Only these models predict a specific class per instance. In Weka, we can get the exact probability for each instance to be of each class. How can we do it using these packages?

在 LogisticRegressionModel 中，我们可以设置阈值.所以我创建了一个函数来检查不同阈值上每个点的结果.但这不能用于 RandomForest(参见 如何在 Spark 随机森林中训练数据时设置截止)

In LogisticRegressionModel we can set the threshold. So I've created a function that check the results for each point on a different threshold. But this cannot be done for RandomForest (see How to set cutoff while training the data in Random Forest in Spark)

推荐答案

不幸的是，在 1.4.1 版本之前，使用 MLLIb 无法获得分类模型的每个实例的概率.

Unfortunately, with MLLIb you can't get the probabilities per instance for classification models till version 1.4.1.

存在 JIRA 问题(SPARK-4362 和 SPARK-6885) 关于这个确切的主题，我现在正在写答案.尽管如此，该问题似乎自 2014 年 11 月起就搁置了

There is JIRA issues (SPARK-4362 and SPARK-6885) concerning this exact topic which is IN PROGRESS as I'm writing the answer now. Nevertheless, the issue seems to be on hold since November 2014

目前无法在预测过程中使用朴素贝叶模型获得预测的后验概率.这应该与标签一起提供.

There is currently no way to get the posterior probability of a prediction with Naive Baye's model during prediction. This should be made available along with the label.

这里是@sean-owen 在邮件列表中关于朴素贝叶斯分类算法的类似主题的注释:

And here is a note from @sean-owen on the mailing list on a similar topic regarding the Naive Bayes classification algorithm:

最近在此邮件列表中讨论了这一点.你现在不能直接得到概率，但你可以稍微修改一下以获得 NaiveBayesModel 的内部数据结构并从那里计算它.

This was recently discussed on this mailing list. You can't get the probabilities out directly now, but you can hack a bit to get the internal data structures of NaiveBayesModel and compute it from there.

参考:源.

主要修改:此问题已在 Spark 1.5.0 中解决.有关详细信息，请参阅 JIRA 问题.

MAJOR EDIT: This issue has been resolved with Spark 1.5.0. Please refer to the JIRA issue for more details.

这篇关于如何在 spark.mllib 中的分类模型中获得每个实例的概率的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！