scikit-learn的TfidfVectorizer的在线版本 [英] Online version of scikit-learn's TfidfVectorizer

问题描述

我希望使用scikit-learn的HashingVectorizer，因为它非常适合在线学习问题(文本中的新标记必定会映射到存储桶").不幸的是，scikit-learn中包含的实现似乎并未包含对tf-idf功能的支持.通过TfidfTransformer传递矢量化器输出是使在线更新与tf-idf功能一起工作的唯一方法，还是那里有更优雅的解决方案?

I'm looking to use scikit-learn's HashingVectorizer because it's a great fit for online learning problems (new tokens in text are guaranteed to map to a "bucket"). Unfortunately the implementation included in scikit-learn doesn't seem to include support for tf-idf features. Is passing the vectorizer output through a TfidfTransformer the only way to make online updates work with tf-idf features, or is there a more elegant solution out there?

推荐答案

本质上，您不能以在线方式使用TF IDF，因为过去的所有功能的IDF都会随每个新文档而变化-这意味着需要重新访问并重新培训所有先前的文档，而这些文档将不再在线.

Intrinsically you can not use TF IDF in an online fashion, as the IDF of all past features will change with every new document - which would mean re-visiting and re-training on all the previous documents, which would no-longer be online.

可能会有一些近似值，但是您必须自己实现它们.

There may be some approximations, but you would have to implement them yourself.

这篇关于scikit-learn的TfidfVectorizer的在线版本的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！