如何将 Spark 中的分类变量转换为一组编码为 {0,1} 的列? [英] How to transform a categorical variable in Spark into a set of columns coded as {0,1}?

问题描述

我正在尝试使用 Spark MLlib(使用 Scala)对包含分类变量的数据集执行逻辑回归 (LogisticRegressionWithLBFGS).我发现 Spark 无法处理这种变量.

I'm trying to perform a logistic regression (LogisticRegressionWithLBFGS) with Spark MLlib (with Scala) on a dataset which contains categorical variables. I discover Spark was not able to work with that kind of variable.

在 R 中有一种简单的方法来处理这种问题:我在因子(类别)中转换变量，因此 R 创建一组编码为 {0,1} 指示变量的列.

In R there is a simple way to deal with that kind of problem : I transform the variable in factor (categories), so R creates a set of columns coded as {0,1} indicator variables.

如何使用 Spark 执行此操作?

How can I perform this with Spark?

推荐答案

使用 VectorIndexer，您可以告诉索引器一个字段可能具有的不同值(基数)的数量，以便使用 setMaxCategories() 方法将其视为分类字段.

Using VectorIndexer, you may tell the indexer the number of different values (cardinality) that a field may have in order to be considered categorical with the setMaxCategories() method.

val indexer = new VectorIndexer()
.setInputCol("features")
.setOutputCol("indexed")
.setMaxCategories(10)

来自 Scaladocs:

用于索引 Vector 数据集中分类特征列的类.

Class for indexing categorical feature columns in a dataset of Vector.

这有两种使用模式:

自动识别分类特征(默认行为)

Automatically identify categorical features (default behavior)

这有助于将未知向量的数据集处理成具有一些连续特征和一些分类特征.连续和分类之间的选择基于 maxCategories 参数.

This helps process a dataset of unknown vectors into a dataset with some continuous features and some categorical features. The choice between continuous and categorical is based upon a maxCategories parameter.

将 maxCategories 设置为任何分类特征应具有的最大分类数.

Set maxCategories to the maximum number of categorical any categorical feature should have.

例如:特征 0 具有唯一值 {-1.0, 0.0}，特征 1 具有唯一值 {1.0, 3.0, 5.0}.如果 maxCategories = 2，则特征 0 将被声明为分类并使用索引 {0, 1}，特征 1 将被声明为连续的.

E.g.: Feature 0 has unique values {-1.0, 0.0}, and feature 1 values {1.0, 3.0, 5.0}. If maxCategories = 2, then feature 0 will be declared categorical and use indices {0, 1}, and feature 1 will be declared continuous.

我发现这是提取分类值的一种方便(虽然粗粒度)的方法，但要注意是否在任何情况下您都有一个想要连续的低数量字段(例如大学生的年龄与原籍国或美国州).

I find this a convenient (though coarse-grained) way to extract the categorical values, but beware if in any case you have a field with lower arity that you want to be continuous (e.g. age in college students vs origin country or US-state).

这篇关于如何将 Spark 中的分类变量转换为一组编码为 {0,1} 的列?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！