大数据集的TFIDF [英] TFIDF for Large Dataset

问题描述

我有一个大约有800万条新闻文章的语料库，我需要以稀疏矩阵的形式获取它们的TFIDF表示形式.我已经能够使用scikit-learn来实现相对较少的样本数量的操作，但是我认为它不能用于如此庞大的数据集，因为它首先将输入矩阵加载到内存中，这是一个昂贵的过程.

I have a corpus which has around 8 million news articles, I need to get the TFIDF representation of them as a sparse matrix. I have been able to do that using scikit-learn for relatively lower number of samples, but I believe it can't be used for such a huge dataset as it loads the input matrix into memory first and that's an expensive process.

有人知道，为大型数据集提取TFIDF向量的最佳方法是什么?

Does anyone know, what would be the best way to extract out the TFIDF vectors for large datasets?

推荐答案

Gensim具有高效的 tf-idf模型，并且不需要一次将所有内容存储在内存中.

Gensim has an efficient tf-idf model and does not need to have everything in memory at once.

您的语料库只需要是可迭代的，因此不需要一次将整个语料库存储在内存中.

Your corpus simply needs to be an iterable, so it does not need to have the whole corpus in memory at a time.

make_wiki脚本在Wikipedia中运行根据评论，在笔记本电脑上大约50m.

The make_wiki script runs over Wikipedia in about 50m on a laptop according to the comments.

这篇关于大数据集的TFIDF的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！