如何将模型从 ML Pipeline 保存到 S3 或 HDFS? [英] How to save models from ML Pipeline to S3 or HDFS?

问题描述

我正在尝试保存 ML Pipeline 生成的数千个模型.如答案中所述此处，模型可以保存如下:

I am trying to save thousands of models produced by ML Pipeline. As indicated in the answer here, the models can be saved as follows:

import java.io._

def saveModel(name: String, model: PipelineModel) = {
  val oos = new ObjectOutputStream(new FileOutputStream(s"/some/path/$name"))
  oos.writeObject(model)
  oos.close
}

schools.zip(bySchoolArrayModels).foreach{
  case (name, model) => saveModel(name, Model)
}

我尝试使用 s3://some/path/$name 和 /user/hadoop/some/path/$name 因为我希望模型能够最终被保存到 amazon s3，但它们都失败了，消息表明找不到路径.

I have tried using s3://some/path/$name and /user/hadoop/some/path/$name as I would like the models to be saved to amazon s3 eventually but they both fail with messages indicating the path cannot be found.

如何将模型保存到 Amazon S3?

How to save models to Amazon S3?

推荐答案

一种将模型保存到 HDFS 的方法如下:

One way to save a model to HDFS is as following:

// persist model to HDFS
sc.parallelize(Seq(model), 1).saveAsObjectFile("hdfs:///user/root/linReg.model")

然后可以将保存的模型加载为:

Saved model can then be loaded as:

val linRegModel = sc.objectFile[LinearRegressionModel]("linReg.model").first()

有关更多详细信息，请参阅 (ref)

For more details see (ref)

这篇关于如何将模型从 ML Pipeline 保存到 S3 或 HDFS?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！