VectorAssembler不支持StringType类型的scala spark转换 [英] VectorAssembler does not support the StringType type scala spark convert

问题描述

我有一个包含字符串列的数据框，我打算将其用作使用spark和scala的k均值的输入.我正在使用以下方法转换数据框的字符串类型的列:

I have a dataframe that contains string columns and I am planning to use it as input for k-means using spark and scala. I am converting my string typed columns of the dataframe using the method below:

 val toDouble = udf[Double, String]( _.toDouble) 
 val analysisData  = dataframe_mysql.withColumn("Event", toDouble(dataframe_mysql("event"))).withColumn("Execution", toDouble(dataframe_mysql("execution"))).withColumn("Info", toDouble(dataframe_mysql("info")))              
 val assembler = new VectorAssembler()
    .setInputCols(Array("execution", "event", "info"))
    .setOutputCol("features")
val output = assembler.transform(analysisData)
println(output.select("features", "execution").first())

当我打印analysisData模式时，转换是正确的.但我遇到一个例外: VectorAssembler不支持StringType类型 这意味着我的值仍然是字符串！我该如何转换值，而不仅仅是模式类型?

when I print the analysisData schema the convertion is correct. but I am getting an exception: VectorAssembler does not support the StringType type which means that my values are still strings! how can I convert the values and not only the schema type?

谢谢

推荐答案

实际上，VectorAssembler转换器不接受字符串.因此，您需要确保您的列与数值，布尔值，向量类型匹配.确保您的udf做正确的事，并确保所有列都不具有StringType.

Indeed, the VectorAssembler Transformer does not take strings. So you need to make sure that your columns match numerical, boolean, vector types. Make sure that your udf is doing the right thing and be sure that none of the columns has StringType.

要将Spark DataFrame中的列转换为另一种类型，请使其变得简单并使用cast()DSL函数，如下所示:

To convert a column in a Spark DataFrame to another type, make it simple and use the cast() DSL function like so:

val analysisData  = dataframe_mysql.withColumn("Event", dataframe_mysql("Event").cast(DoubleType))

应该可以！

这篇关于VectorAssembler不支持StringType类型的scala spark转换的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！