sparkml setParallelism用于交叉验证 [英] sparkml setParallelism for crossvalidator
本文介绍了sparkml setParallelism用于交叉验证的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
所以我尝试使用SparkML设置交叉验证,但出现运行时错误提示
so I am trying to set a cross validation using SparkML but I am getting a run time error saying that
"value setParallelism is not a member of org.apache.spark.ml.tuning.CrossValidator"
我目前正在关注Spark页面教程.我对此并不陌生,因此不胜感激.贝娄是我的代码段:
I am currently following the spark page tutorial. I am new to this so any help is appreciated. Bellow is my code snippet:
import org.apache.spark.ml.{Pipeline, PipelineModel}
import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.ml.feature.{HashingTF, Tokenizer}
import org.apache.spark.ml.linalg.Vector
import org.apache.spark.sql.Row
import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
import org.apache.spark.ml.tuning.{CrossValidator, ParamGridBuilder}
// Tokenizer
val tokenizer = new Tokenizer().setInputCol("tweet").setOutputCol("words")
// HashingTF
val hash_tf = new HashingTF().setInputCol(tokenizer.getOutputCol).setOutputCol("features")
// ML models
val l_regression = new LogisticRegression().setMaxIter(100).setRegParam(0.15)
// Pipeline
val pipe = new Pipeline().setStages(Array(tokenizer, hash_tf, l_regression))
val paramGrid = new ParamGridBuilder()
.addGrid(hash_tf.numFeatures, Array(10,100,1000))
.addGrid(l_regression.regParam, Array(0.1,0.01,0.001))
.build()
val c_validator = new CrossValidator()
.setEstimator(pipe)
.setEvaluator(new BinaryClassificationEvaluator)
.setEstimatorParamMaps(paramGrid)
.setNumFolds(3)
.setParallelism(2)
推荐答案
setParallelism
is available only in Spark 2.3 or later. You must be using earlier version:
(仅限专家)参数设置器
(expert-only) Parameter setters
(...)
def setParallelism(value: Int): CrossValidator.this.type
设置最大并行度,以并行评估模型.串行评估默认为1
Set the maximum level of parallelism to evaluate models in parallel. Default is 1 for serial evaluation
注释@Since("2.3.0")
Annotations @Since( "2.3.0" )
这篇关于sparkml setParallelism用于交叉验证的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文