将元数据附加到Spark中的向量列 [英] Attach metadata to vector column in Spark
问题描述
上下文: 我有一个包含两列的数据框:标签和要素.
Context: I have a data frame with two columns: label, and features.
org.apache.spark.sql.DataFrame = [label: int, features: vector]
其中 features 是使用VectorAssembler构建的数字类型的mllib.linalg.VectorUDT.
Where features is a mllib.linalg.VectorUDT of numeric type built using VectorAssembler.
问题: 有没有一种将模式分配给特征向量的方法?我想跟踪每个功能的名称.
Question: Is there a way to assign a schema to the features vector? I want to keep track of the name of each feature.
到目前为止已尝试:
val defaultAttr = NumericAttribute.defaultAttr
val attrs = Array("feat1", "feat2", "feat3").map(defaultAttr.withName)
val attrGroup = new AttributeGroup("userFeatures", attrs.asInstanceOf[Array[Attribute]])
scala> attrGroup.toMetadata
res197: org.apache.spark.sql.types.Metadata = {"ml_attr":{"attrs":{"numeric":[{"idx":0,"name":"f1"},{"idx":1,"name":"f2"},{"idx":2,"name":"f3"}]},"num_attrs":3}}
但是不确定如何将其应用于现有数据框.
But was not sure how to apply this to an existing data frame.
推荐答案
至少有两个选择:
-
在现有的
DataFrame
上,可以将as
方法与metadata
参数一起使用:
On existing
DataFrame
you can useas
method withmetadata
argument:
import org.apache.spark.ml.attribute._
val rdd = sc.parallelize(Seq(
(1, Vectors.dense(1.0, 2.0, 3.0))
))
val df = rdd.toDF("label", "features")
df.withColumn("features", $"features".as("_", attrGroup.toMetadata))
创建新的DataFrame
时,转换AttributeGroup
toStructField
并将其用作给定列的架构:
When you create new DataFrame
convert AttributeGroup
toStructField
and use it as a schema for a given column:
import org.apache.spark.sql.types.{StructType, StructField, IntegerType}
val schema = StructType(Array(
StructField("label", IntegerType, false),
attrGroup.toStructField()
))
spark.createDataFrame(
rdd.map(row => Row.fromSeq(row.productIterator.toSeq)),
schema)
如果已使用VectorAssembler
列元数据创建了向量列,则描述父列的列元数据应已附加.
If vector column has been created using VectorAssembler
column metadata describing parent columns should be already attached.
import org.apache.spark.ml.feature.VectorAssembler
val raw = sc.parallelize(Seq(
(1, 1.0, 2.0, 3.0)
)).toDF("id", "feat1", "feat2", "feat3")
val assembler = new VectorAssembler()
.setInputCols(Array("feat1", "feat2", "feat3"))
.setOutputCol("features")
val dfWithMeta = assembler.transform(raw).select($"id", $"features")
dfWithMeta.schema.fields(1).metadata
// org.apache.spark.sql.types.Metadata = {"ml_attr":{"attrs":{"numeric":[
// {"idx":0,"name":"feat1"},{"idx":1,"name":"feat2"},
// {"idx":2,"name":"feat3"}]},"num_attrs":3}
不能使用点语法(例如$features.feat1
)直接访问向量字段,但可以使用VectorSlicer
这样的专用工具使用这些字段:
Vector fields are not directly accessible using dot syntax (like $features.feat1
) but can used by specialized tools like VectorSlicer
:
import org.apache.spark.ml.feature.VectorSlicer
val slicer = new VectorSlicer()
.setInputCol("features")
.setOutputCol("featuresSubset")
.setNames(Array("feat1", "feat3"))
slicer.transform(dfWithMeta).show
// +---+-------------+--------------+
// | id| features|featuresSubset|
// +---+-------------+--------------+
// | 1|[1.0,2.0,3.0]| [1.0,3.0]|
// +---+-------------+--------------+
有关PySpark,请参见如何在XML框架中将列声明为DataFrame中的分类特征
For PySpark see How can I declare a Column as a categorical feature in a DataFrame for use in ml
这篇关于将元数据附加到Spark中的向量列的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!