使用新数据测试文本分类ML模型失败 [英] Testing text classification ML model with new data fails

问题描述

我建立了一个机器学习模型来将电子邮件分类为垃圾邮件.现在，我想测试自己的电子邮件并查看结果.因此，我编写了以下代码对新电子邮件进行分类:

  message ="，主题:您好，这是来自Google安全团队，我们希望恢复您的密码.请联系我们尽快消息= pd.Series([消息，])transform_message = CountVectorizer(分析器= process_text).fit_transform(消息)proba = model.predict_proba(transformed_message)[0] 

知道 process_text 是处理电子邮件的功能，当我运行代码时，出现以下错误:

 模型的特征数量必须与输入匹配.型号n_features为37229，输入n_features为13 

有什么问题，请问如何解决?

解决方案

对于此类管道中的所有数据预处理步骤，我们不再 fit ，就像您在此处所做的那样您的(新定义的)计数向量化器.

因此，不要通过新的计数向量器使用 fit_transform ，而是应通过应用其 transform 方法.这样一来，您的新数据就可以与训练数据的37229特征(模型进行了训练)相对应地映射，而不是在将计数向量化器再次适合如此短的文本时生成的仅13个特征.

I have built a machine learning model to classify emails as spams or not. Now i want to test my own email and see the result. So i wrote the following code to classify the new email:

message = """Subject: Hello this is from google security team we want to recover your password. Please contact us 
as soon as possible"""

message = pd.Series([message,])
transformed_message = CountVectorizer(analyzer=process_text).fit_transform(message)
proba = model.predict_proba(transformed_message)[0]

Knowing that process_text is a function to process the email, When I run the code i get the following error:

Number of features of the model must match the input. Model n_features is 37229 and input n_features is 13 

What's the problem and how can i fix that please ?

解决方案

For all data preprocessing steps in such pipelines, we never fit again, as you do here with your (newly defined) count vectorizer.

So, instead of using fit_transform with a new count vectorizer, you should reuse the existing count vectorizer (i.e. the one used with your training data), by applying its transform method. That will allow your new data to be mapped in relation to the 37229 features of the training data (with which the model was trained), instead of the only 13 features produced when you fit again a count vectorizer to such a short text.

这篇关于使用新数据测试文本分类ML模型失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！