在火花中选择特征后，使测试数据的特征与火车数据相同 [英] Making the features of test data same as train data after featureselection in spark

问题描述

我正在研究Scala.我有一个大问题，ChiSqSelector似乎可以成功地缩小尺寸，但是我无法确定哪些功能被缩小了，剩下的仍然是什么.我怎么知道减少了哪些功能?

I m working on Scala. I have a big question, ChiSqSelector seems to reduce dimension successfully, but I can't identify what features were reduced what were remained. How can I know what features were reduced?

[WrappedArray(a, b, c),(5,[1,2,3],[1,1,1]),(2,[0],[1])]
[WrappedArray(b, d, e),(5,[0,2,4],[1,1,2]),(2,[1],[2])]
[WrappedArray(a, c, d),(5,[0,1,3],[1,1,1]),(2,[0],[1])]

PS:当我想使测试数据与功能选择的火车数据相同时，我发现我不知道如何在Scala中做到这一点.

PS: when I wanted to make the test data same as feature-selected train data I found that I dont know how to do that in scala.

推荐答案

如果使用ChiSqSelector的 MLlib 版本，则可以selectedFeatures:

If you use MLlib version of the ChiSqSelector you can selectedFeatures:

mllibModel: org.apache.spark.mllib.feature.ChiSqSelectorModel = ???
val features: Array[Int] = mllib.selectedFeatures

尽管如此，当您使用测试数据时，最好使用在训练数据集中训练的选择器，而不必理会手动选择.

Nevertheless when you work with test data it is better to use selector trained on the train dataset and don't bother with manual selection.

val testData: RDD[org.apache.spark.mllib.linalg.Vector] = ???
mllibModel.transform(testData)

相同的规则适用于 ML 版本.您可以使用selectedFeatures提取索引数组:

The same rules apply to ML version. You can use selectedFeatures to extract array of indices:

val mlModel: org.apache.spark.ml.feature.ChiSqSelectorModel = ???
val features: Array[Int] = mlModel.selectedFeatures

但是最好保留模型并在新数据上重用:

but it is still better to keep model and reuse on new data:

val testData: RDD[org.apache.spark.sql.DataFrame] = ???
mlModel.transform(testData)

如果您希望获得人类可读的功能列表，则可以在转换后分析列元数据，如在Spark中将列标记为分类" a>

If you want a human readable list of features you can analyze column metadata after transformation as shown in Tagging columns as Categorical in Spark

这篇关于在火花中选择特征后，使测试数据的特征与火车数据相同的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！