如何在 Apache Spark 中保存和加载 MLLib 模型? [英] How to save and load MLLib model in Apache Spark?

问题描述

我在 Apache Spark 中训练了一个分类模型(使用 pyspark).我将模型存储在一个对象中，LogisticRegressionModel.现在，我想对新数据进行预测.我想存储模型，并将其读回新程序以进行预测.知道如何存储模型吗?我正在考虑使用泡菜，但我是 Python 和 Spark 的新手，所以我想听听社区的想法.

I trained a classification model in Apache Spark (using pyspark). I stored the model in an object, LogisticRegressionModel. Now, I want to make predictions on new data. I would like to store the model, and read it back into a new program in order to make the predictions. Any idea how to store the model? I'm thinking of maybe pickle, but I'm a newbie to both python and Spark, so I'd like to hear what the community thinks.

推荐答案

您可以使用 保存方法 mllib 模型.

You can save your model by using the save method of mllib models.

# let lrm be a LogisticRegression Model
lrm.save(sc, "lrm_model.model")

存储后，您可以将其加载到另一个应用程序中.

After storing it you can load it in another application.

sameModel = LogisticRegressionModel.load(sc, "lrm_model.model")

正如@zero323 之前所述，还有另一种方法可以实现这一点，即使用 预测模型标记语言 (PMML).

As @zero323 stated before, there is another way to achieve this, and is by using the Predictive Model Markup Language (PMML).

是由 Data Mining Group 开发的一种基于 XML 的文件格式，为应用程序提供一种方法来描述和交换由数据挖掘和机器学习算法生成的模型.

is an XML-based file format developed by the Data Mining Group to provide a way for applications to describe and exchange models produced by data mining and machine learning algorithms.

这篇关于如何在 Apache Spark 中保存和加载 MLLib 模型?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！