如何让 VectorAssembler 不压缩数据? [英] How to make VectorAssembler do not compress data?

问题描述

我想使用 VectorAssembler 将多列转换为一列，但默认情况下数据是压缩的，没有其他选项.

I want to transform multiple columns to one column using VectorAssembler,but the data is compressed by default without other options.

val arr2= Array((1,2,0,0,0),(1,2,3,0,0),(1,2,4,5,0),(1,2,2,5,6))
val df=sc.parallelize(arr2).toDF("a","b","c","e","f")
val colNames=Array("a","b","c","e","f")
val assembler = new VectorAssembler()
  .setInputCols(colNames)
  .setOutputCol("newCol")
val transDF= assembler.transform(df).select(col("newCol"))
transDF.show(false)

输入是:

  +---+---+---+---+---+
  |  a|  b|  c|  e|  f|
  +---+---+---+---+---+
  |  1|  2|  0|  0|  0|
  |  1|  2|  3|  0|  0|
  |  1|  2|  4|  5|  0|
  |  1|  2|  2|  5|  6|
  +---+---+---+---+---+

结果是:

+---------------------+
|newCol               |
+---------------------+
|(5,[0,1],[1.0,2.0])  |
|[1.0,2.0,3.0,0.0,0.0]|
|[1.0,2.0,4.0,5.0,0.0]|
|[1.0,2.0,2.0,5.0,6.0]|
+---------------------+

我期望的结果是:

+---------------------+
|newCol               |
+---------------------+
|[1.0,2.0,0.0,0.0,0.0]|
|[1.0,2.0,3.0,0.0,0.0]|
|[1.0,2.0,4.0,5.0,0.0]|
|[1.0,2.0,2.0,5.0,6.0]|
+---------------------+

我应该怎么做才能得到预期的结果?

What should I do to get my expect result?

推荐答案

如果你真的想将所有向量强制转换为它们的密集表示，你可以使用用户定义函数来实现:

If you really want to coerce all vectors to their dense representation, you can do it using a User Defined Function :

val toDense = udf((v: org.apache.spark.ml.linalg.Vector) => v.toDense)
transDF.select(toDense($"newCol")).show

+--------------------+
|         UDF(newCol)|
+--------------------+
|[1.0,2.0,0.0,0.0,...|
|[1.0,2.0,3.0,0.0,...|
|[1.0,2.0,4.0,5.0,...|
|[1.0,2.0,2.0,5.0,...|
+--------------------+

这篇关于如何让 VectorAssembler 不压缩数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！