为 sklearn 管道实现自定义单热编码功能 [英] implement custom one-hot-encoding function for sklearn pipeline

问题描述

与 一个热编码保留 NA插补 我正在尝试创建一个自定义函数，用于在对分类变量进行热编码时处理 NA.该设置应适合使用 sklearn 管道 进行训练/测试拆分和建模.

In related to question posted in One Hot Encoding preserve the NAs for imputation I am trying to create a custom function that handles NAs when one hot encoding categorical variables. The setup should be suitable for train/test split and modelling using sklearn pipeline.

我的问题的一个简单的可重现示例:

A simple reproducible example of my problem:

#Packages
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.impute import KNNImputer
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.linear_model import Ridge
from sklearn.impute import SimpleImputer


# Make some categorical data X and a response y and split it. 
X = pd.DataFrame(columns=["1","2"],data = [["A",np.nan],["B","A"],[np.nan,"A"],[np.nan,"B"],["B","A"],["A","B"],["C","B"],["D","E"]])
y = pd.DataFrame(data = np.array([1,5,4,6,2,3,9,9]))
X_train, X_test, Y_train, Y_test = train_test_split(X,y,test_size=0.2,random_state=42)

然后我创建了一个使用 nan 执行 OHE 的自定义函数(使用 Scikit-learn 中 OneHotEncoder 和 KNNImpute 之间的循环)

I've then created a custom function that does OHE with nan (using the procedure described in Cyclical Loop Between OneHotEncoder and KNNImpute in Scikit-learn)

class OHE_with_nan(BaseEstimator,TransformerMixin):
    """ OHE with NAN. Not super pretty but works..
    """
    def __init__(self, copy=True):
        self.copy = copy
        
    def fit(self, X, y = None):
        """ This transformer does not use a fit procedure """
        return self
    
    def transform(self, X, y = None):
        """ Return the new object here"""
        # Replace nans with "Missing" such that OneHotEncoder can work.
        enc_missing = SimpleImputer(strategy="constant",fill_value="missing")
        data1 = pd.DataFrame(columns=X.columns,data = enc_missing.fit_transform(X))
        #Perform standard OHE
        OHE = OneHotEncoder(sparse=False,handle_unknown="ignore")
        OHE_fit = OHE.fit_transform(data1)
        #save feature names of the OHE dataframe
        data_OHE = pd.DataFrame(columns=OHE.get_feature_names(data1.columns),data = OHE_fit)
        
        # Initialize
        Column_names = data1.columns
        Final_OHE = pd.DataFrame()
        # Loop over columns to replace 0s with nan the correct places.
        for i in range(len(data1.columns)):
           tmp_data = data_OHE[data_OHE.columns[pd.Series(data_OHE.columns).str.startswith(Column_names[i])]]
           missing_name = tmp_data.iloc[:,-1:].columns
           missing_index = np.where(tmp_data[missing_name]==1)[0]
           tmp_data.loc[missing_index,:] = np.nan
           tmp_data1 = tmp_data.drop(missing_name,axis=1)
           Final_OHE = pd.concat([Final_OHE, tmp_data1], axis=1)
        
        return Final_OHE

然后将其组合成一个使用岭回归预测 y 的管道(模型的随机选择，仅用于示例..)

This is then combined into a pipeline that predicts y using ridge regression (random choice of model, just for the example..)

Estimator = Pipeline([
   ('Ohe_with_NA',OHE_with_nan()),
   ("Imputer",KNNImputer(n_neighbors=1)),
   ('Model',Ridge(alpha = 0.01))
    ])

可以拟合的程序:

pipe_fit = Estimator.fit(X_train,Y_train)

但是对看不见的数据的测试失败了:

But testing on unseen data fails:

pipe_fit.score(X_test, Y_test)

ValueError: X has 2 features, but KNNImputer is expecting 7 features as input.

这是因为 OHE_with_nan 内的 OneHotEncoder 中的 handle_unknown = "ignore 不再活动"；因为它已被包装到我的自定义函数中.

This is because the handle_unknown = "ignore in OneHotEncoder within OHE_with_nanis no longer "active" as it has been wrapped into my custom function.

如果只是在管道中直接使用 OneHotEncoder(handle_unknown = "ignore")，一切正常(但这不是我的意图，因为这从我尝试的数据中删除"了 nans推算.)

If one simply uses OneHotEncoder(handle_unknown = "ignore") directly in the pipeline, everything works fine (but that's not my intention as this "removes" the nans from the data I try to impute.)

我的问题如何在我的自定义函数中启用 handle_unknown = "ignore" 以便它也可以在未见数据的管道设置中执行?

My question How do I enable handle_unknown = "ignore" in my custom function such that it can perform in a pipeline setup on unseen data as well?

希望您了解我的情况 - 非常感谢您的帮助！

Hope you understand my situation - any help is highly appreciated!

推荐答案

我认为主要的问题是你需要在合适的时候保存更多的信息(尤其是内部的OneHotEncoder).我还使缺失列识别更加强大(我想也许您依赖于将其放在最后的排序，但由于字母顺序，这仅适用于您的样本数据?).我没有花太多时间清理东西或寻求效率.

I think the main problem is that you need to save more information (especially, the internal OneHotEncoder) at fit time. I also made the missing-column identification a little more robust (I think maybe you were relying on the ordering putting that last, but that only held for your sample data because of alphabetical order?). I didn't spend much time cleaning things up or looking for efficiencies.

class OHE_with_nan(BaseEstimator, TransformerMixin):
    """One-hot encode, propagating NaNs.

    Requires a dataframe as input!
    """
    def fit(self, X, y=None):
        self.orig_cols_ = X.columns
        self.imputer_ = SimpleImputer(strategy="constant", fill_value="MISSING")
        X_filled = self.imputer_.fit_transform(X)
        self.ohe_ = OneHotEncoder(sparse=False, handle_unknown="ignore")
        self.ohe_.fit(X_filled)

        self.ohe_colnames_ = self.ohe_.get_feature_names(X.columns)
        self.missing_value_columns = np.array(["MISSING" in col for col in self.ohe_colnames_])
        return self

    def transform(self, X, y=None):
        raw_ohe = pd.DataFrame(self.ohe_.transform(self.imputer_.transform(X)), columns=self.ohe_colnames_)
        out_list = []
        # Loop over columns to replace 0s with nan the correct places.
        for orig_col in self.orig_cols_:
            tmp_data = raw_ohe[self.ohe_colnames_[pd.Series(self.ohe_colnames_).str.startswith(orig_col)]]
            missing_name = tmp_data.columns[["MISSING" in col for col in tmp_data.columns]]
            missing_indices = np.where(tmp_data[missing_name]==1)[0]
            tmp_data.loc[missing_indices, :] = np.nan
            tmp_data1 = tmp_data.drop(missing_name, axis=1)
            out_list.append(tmp_data1)
        out = pd.concat(out_list, axis=1)
        return out

这篇关于为 sklearn 管道实现自定义单热编码功能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！