大数据集的 TFIDF [英] TFIDF for Large Dataset

问题描述

我有一个包含大约 800 万篇新闻文章的语料库，我需要将它们的 TFIDF 表示为稀疏矩阵.我已经能够将 scikit-learn 用于相对较少数量的样本，但我相信它不能用于如此庞大的数据集，因为它首先将输入矩阵加载到内存中，这是一个昂贵的过程.

I have a corpus which has around 8 million news articles, I need to get the TFIDF representation of them as a sparse matrix. I have been able to do that using scikit-learn for relatively lower number of samples, but I believe it can't be used for such a huge dataset as it loads the input matrix into memory first and that's an expensive process.

有谁知道提取大型数据集的 TFIDF 向量的最佳方法是什么?

Does anyone know, what would be the best way to extract out the TFIDF vectors for large datasets?

推荐答案

Gensim 有一个高效的 tf-idf 模型 并且不需要一次将所有内容都保存在内存中.

Gensim has an efficient tf-idf model and does not need to have everything in memory at once.

你的语料库只需要是一个可迭代的，所以它不需要一次在内存中存储整个语料库.

Your corpus simply needs to be an iterable, so it does not need to have the whole corpus in memory at a time.

make_wiki 脚本在维基百科上运行了大约根据评论在笔记本电脑上 50m.

The make_wiki script runs over Wikipedia in about 50m on a laptop according to the comments.

这篇关于大数据集的 TFIDF的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！