如何在Apache Spark Pipeline中打印最佳模型参数? [英] How to print best model params in Apache Spark Pipeline?

问题描述

我正在使用Apache Spark的管道API进行参数验证. 我正在像这样构建TrainValidationSplitModel:

I'm using pipeline API of Apache Spark for validation of parameters. I'm building TrainValidationSplitModel like this :

Pipeline pipeline = ...
ParamMap[] paramGrid = ...

TrainValidationSplit trainValidationSplit = new TrainValidationSplit().setEstimator(pipeline).setEvaluator(new MulticlassClassificationEvaluator()).setEstimatorParamMaps(paramGrid).setTrainRatio(0.8);
TrainValidationSplitModel model = trainValidationSplit.fit(training);

我的问题是:如何提取和打印最佳训练模型的参数?

My question is: how can I extract and print params of best trained model?

推荐答案

最后我做到了. 培训后，Spark会打印此指标.我的火花具有错误日志级别，所以我没有看到以下内容:

Finally I did it. Spark prints this metrics after training. I had ERROR log level for spark, so I haven't seen this:

2015-10-21 12:57:33,828 [INFO  org.apache.spark.ml.tuning.TrainValidationSplit]
Train validation split metrics: WrappedArray(0.7141940371838821, 0.7358721053749735)

2015-10-21 12:57:33,831 [INFO  org.apache.spark.ml.tuning.TrainValidationSplit]
Best set of parameters:
{
    hashingTF_79cf758f5ab1-numFeatures: 2000000,
    nb_67d55ce4e1fc-smoothing: 1.0
}

2015-10-21 12:57:33,831 [INFO  org.apache.spark.ml.tuning.TrainValidationSplit]
Best train validation split metric: 0.7358721053749735.

现在，我在log4j.properties文件中为TrainValidationSplit类添加了级别INFO:

Now I've added level INFO for class TrainValidationSplit in my log4j.properties file:

log4j.logger.org.apache.spark.ml.tuning.TrainValidationSplit=INFO
log4j.additivity.org.apache.spark.ml.tuning.TrainValidationSplit=false

这篇关于如何在Apache Spark Pipeline中打印最佳模型参数?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！