在训练和测试集中具有不同级别的管道中的虚拟创建 [英] Dummy creation in pipeline with different levels in train and test set

问题描述

我目前正在探索 scikit 学习管道.我还想用管道预处理数据.但是，我的训练和测试数据具有不同级别的分类变量.例子:考虑:

I'm currently exploring the scikit learn pipelines. I also want to preprocess the data with a pipeline. However, my train and test data have different levels of the categorical variable. Example: Consider:

import pandas as pd
train = pd.Series(list('abbaa'))
test = pd.Series(list('abcd'))

我用熊猫写了一个 TransformerMixinClass

I wrote a TransformerMixinClass using pandas

class CreateDummies(TransformerMixin):

def transform(self, X, **transformparams):
    return pd.get_dummies(X).copy()

def fit(self, X, y=None, **fitparams):
    return self

fit_transform 产生 2 列训练数据和 4 列测试数据.所以这里并不奇怪，但不适合管道

fit_transform yields for the train data 2 columns and for the test data 4 columns. So no surprise here, but not suitable for a pipeline

同样，我尝试导入标签编码器(以及潜在的后续步骤的 OneHotEncoder):

Similary, I tried to import the label encoder (and OneHotEncoder for the potential next steps):

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
le = LabelEncoder()
le.fit_transform(train)
le.transform(test)

这会产生一个错误，这并不奇怪.

which yields, not surprisingly, an error.

所以这里的问题是我需要一些包含在测试集中的信息.有什么好的方法可以将其包含在管道中吗?

So the problem here is that I need some information contained in the test set. Is there a good way to include this in a pipeline?

推荐答案

您可以使用 this answer 中所述的分类词:

You can use categoricals as explained in this answer:

categories = np.union1d(train, test)
train = train.astype('category', categories=categories)
test = test.astype('category', categories=categories)

pd.get_dummies(train)
Out: 
   a  b  c  d
0  1  0  0  0
1  0  1  0  0
2  0  1  0  0
3  1  0  0  0
4  1  0  0  0

pd.get_dummies(test)
Out: 
   a  b  c  d
0  1  0  0  0
1  0  1  0  0
2  0  0  1  0
3  0  0  0  1

这篇关于在训练和测试集中具有不同级别的管道中的虚拟创建的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！