如何使用sklearn Pipeline&选择多个(数字和文本)列FeatureUnion用于文本分类? [英] How to select multiple (numerical & text) columns using sklearn Pipeline & FeatureUnion for text classification?
问题描述
我已经开发了用于多标签分类的文本模型. OneVsRestClassifier LinearSVC模型使用sklearns Pipeline和FeatureUnion用于模型准备.
I have developed a text model for multilabel classification. The OneVsRestClassifier LinearSVC model uses sklearns Pipeline and FeatureUnion for model preparation.
主要输入功能包括一个名为response的文本列，以及一个由t1_prob-t5_prob预测的5个可能标签的5个主题概率(从以前的LDA主题模型生成).生成TfidfVectorizer的管道中还有其他功能创建步骤.
The primary input features consist of a text column called response but also 5 topic probabilities (generated from a previous LDA Topic Model) called t1_prob - t5_prob to predict the 5 possible labels. There are other feature creation steps in the pipeline for the the generation of the TfidfVectorizer.
我最终用 ItemSelector 调用了每一列，然后执行ArrayCaster(有关这些函数的定义，请参见下面的代码)分别在这些主题概率列上重复5次.有没有更好的方法来使用 FeatureUnion 选择管道中的多个列? (因此我不必做5次)
I ended up calling each column with ItemSelector and performing the ArrayCaster (see the code below for function definition) 5 times on these topic probability columns individually. Is there a better way to use FeatureUnion to select multiple columns in a pipeline? (so I don't have to do it 5 times)
我想知道是否有必要复制topic1_feature-topic5_feature代码，或者是否可以更简洁地选择多个列?
I am wondering if it is necessary to duplicate the topic1_feature -topic5_feature code or if multiple columns can be selected in a more concise way?
我要输入的数据是Pandas dataFrame:
The data I am feeding in is a Pandas dataFrame:
id response label_1 label_2 label3 label_4 label_5 t1_prob t2_prob t3_prob t4_prob t5_prob
1 Text from response... 0.0 0.0 0.0 0.0 0.0 0.0 0.0625 0.0625 0.1875 0.0625 0.1250
2 Text to model with... 0.0 0.0 0.0 0.0 0.0 0.0 0.1333 0.1333 0.0667 0.0667 0.0667
3 Text to work with ... 0.0 0.0 0.0 0.0 0.0 0.0 0.1111 0.0938 0.0393 0.0198 0.2759
4 Free text comments ... 0.0 0.0 1.0 1.0 0.0 0.0 0.2162 0.1104 0.0341 0.0847 0.0559
x_train是response，并且有5个主题概率列(t1_prob，t2_prob，t3_prob，t4_prob，t5_prob).
The x_train is response and the 5 topic probability columns (t1_prob, t2_prob, t3_prob, t4_prob, t5_prob).
y_train是5个label列，我在其中调用了.values来返回DataFrame的numpy表示形式. (label_1，label_2，label3，label_4，label_5)
The y_train is the 5 label columns which I have called .values on to return a numpy representation of the DataFrame. (label_1, label_2, label3, label_4, label_5)
示例数据框:
import pandas as pd
column_headers = ["id", "response",
"label_1", "label_2", "label3", "label_4", "label_5",
"t1_prob", "t2_prob", "t3_prob", "t4_prob", "t5_prob"]
input_data = [
[1, "Text from response",0.0,0.0,1.0,0.0,0.0,0.0625,0.0625,0.1875,0.0625,0.1250],
[2, "Text to model with",0.0,0.0,0.0,0.0,0.0,0.1333,0.1333,0.0667,0.0667,0.0667],
[3, "Text to work with",0.0,0.0,0.0,0.0,0.0,0.1111,0.0938,0.0393,0.0198,0.2759],
[4, "Free text comments",0.0,0.0,1.0,1.0,1.0,0.2162,0.1104,0.0341,0.0847,0.0559]
]
df = pd.DataFrame(input_data, columns = column_headers)
df = df.set_index('id')
df
我认为我的实现有点麻烦，因为FeatureUnion在组合它们时只能处理二维数组，因此其他任何类型(如DataFrame)对我来说都是成问题的.但是，此示例有效-我只是在寻找改进它并使它更干燥的方法.
I think my implementation is a little bit round about because FeatureUnion will only handle 2-D arrays when combining them, so any other type like DataFrame have been problematic for me. However, this example works--I am just looking for ways to improve it and make it more DRY.
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.base import BaseEstimator, TransformerMixin
class ItemSelector(BaseEstimator, TransformerMixin):
def __init__(self, column):
self.column = column
def fit(self, X, y=None):
return self
def transform(self, X, y=None):
return X[self.column]
class ArrayCaster(BaseEstimator, TransformerMixin):
def fit(self, x, y=None):
return self
def transform(self, data):
return np.transpose(np.matrix(data))
def basic_text_model(trainX, testX, trainY, testY, classLabels, plotPath):
'''OneVsRestClassifier for multi-label prediction'''
pipeline = Pipeline([
('features', FeatureUnion([
('topic1_feature', Pipeline([
('selector', ItemSelector(column='t1_prob')),
('caster', ArrayCaster())
])),
('topic2_feature', Pipeline([
('selector', ItemSelector(column='t2_prob')),
('caster', ArrayCaster())
])),
('topic3_feature', Pipeline([
('selector', ItemSelector(column='t3_prob')),
('caster', ArrayCaster())
])),
('topic4_feature', Pipeline([
('selector', ItemSelector(column='t4_prob')),
('caster', ArrayCaster())
])),
('topic5_feature', Pipeline([
('selector', ItemSelector(column='t5_prob')),
('caster', ArrayCaster())
])),
('word_features', Pipeline([
('vect', CountVectorizer(analyzer="word", stop_words='english')),
('tfidf', TfidfTransformer(use_idf = True)),
])),
])),
('clf', OneVsRestClassifier(svm.LinearSVC(random_state=random_state)))
])
# Fit the model
pipeline.fit(trainX, trainY)
predicted = pipeline.predict(testX)
My incorporation of ArrayCaster into the process arose from this answer.
推荐答案
我使用问题.修改后的管道更加简洁.
I figured out the answer to this question using the FunctionTransformer inspired by @Marcus V's solution to this question. The revised pipeline is much more succinct.
from sklearn.preprocessing import FunctionTransformer
get_numeric_data = FunctionTransformer(lambda x: x[['t1_prob', 't2_prob', 't3_prob', 't4_prob', 't5_prob']], validate=False)
pipeline = Pipeline([
('features', FeatureUnion([
('numeric_features', Pipeline([
('selector', get_numeric_data)
])),
('word_features', Pipeline([
('vect', CountVectorizer(analyzer="word", stop_words='english')),
('tfidf', TfidfTransformer(use_idf = True)),
])),
])),
('clf', OneVsRestClassifier(svm.LinearSVC(random_state=random_state)))
])
这篇关于如何使用sklearn Pipeline&选择多个(数字和文本)列FeatureUnion用于文本分类?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
