在 scikit-learn Pipeline 中获取中间数据状态 [英] Get intermediate data state in scikit-learn Pipeline
问题描述
给出以下示例:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import NMF
from sklearn.pipeline import Pipeline
import pandas as pd
pipe = Pipeline([
("tf_idf", TfidfVectorizer()),
("nmf", NMF())
])
data = pd.DataFrame([["Salut comment tu vas", "Hey how are you today", "I am okay and you ?"]]).T
data.columns = ["test"]
pipe.fit_transform(data.test)
我想在 scikit 学习管道中获取与 tf_idf 输出(在 tf_idf 上的 fit_transform 之后,但不是 NMF 输入)或 NMF 输入对应的中间数据状态.或者换个说法,就跟申请一样了
I would like to get intermediate data state in scikit learn pipeline corresponding to tf_idf output (after fit_transform on tf_idf but not NMF) or NMF input. Or to say things in another way, it would be the same than to apply
TfidfVectorizer().fit_transform(data.test)
我知道 pipe.named_steps["tf_idf"] 可以得到中间变换器,但是我无法得到数据,只能用这种方法得到变换器的参数.
I know pipe.named_steps["tf_idf"] ti get intermediate transformer, but I can't get data, only parameters of the transformer with this method.
推荐答案
正如@Vivek Kumar 在评论中所建议的和我回答的 here,我发现打印信息或将中间数据帧写入 csv 的调试步骤很有用:
As @Vivek Kumar suggested in the comment and as I answered here, I find a debug step that prints information or writes intermediate dataframes to csv useful:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import NMF
from sklearn.pipeline import Pipeline
import pandas as pd
from sklearn.base import TransformerMixin, BaseEstimator
class Debug(BaseEstimator, TransformerMixin):
def transform(self, X):
print(X.shape)
self.shape = shape
# what other output you want
return X
def fit(self, X, y=None, **fit_params):
return self
pipe = Pipeline([
("tf_idf", TfidfVectorizer()),
("debug", Debug()),
("nmf", NMF())
])
data = pd.DataFrame([["Salut comment tu vas", "Hey how are you today", "I am okay and you ?"]]).T
data.columns = ["test"]
pipe.fit_transform(data.test)
编辑
我现在向调试转换器添加了一个状态.现在您可以使用@datasailor 的答案访问形状:
Edit
I now added a state to the debug transformer. Now you can access the shape as in the answer by @datasailor with:
pipe.named_steps["debug"].shape
这篇关于在 scikit-learn Pipeline 中获取中间数据状态的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!