标记化文本中ngram(字符串)的出现频率 [英] Frequency of ngrams (strings) in tokenized text
问题描述
我有一组唯一的ngram(称为ngramlist的列表)和ngram标记化的文本(称为ngrams的列表).我想构造一个新的向量freqlist,其中freqlist的每个元素都是ngram的分数,它等于ngramlist的该元素.我写了下面的代码,给出了正确的输出,但是我想知道是否有一种优化它的方法:
I have a set of unique ngrams (list called ngramlist) and ngram tokenized text (list called ngrams). I want to construct a new vector, freqlist, where each element of freqlist is the fraction of ngrams that is equal to that element of ngramlist. I wrote the following code that gives the correct output, but I wonder if there is a way to optimize it:
freqlist = [
sum(int(ngram == ngram_condidate)
for ngram_condidate in ngrams) / len(ngrams)
for ngram in ngramlist
]
我想nltk或其他地方有一个函数可以更快地完成此操作,但是我不确定哪个函数.
I imagine there is a function in nltk or elsewhere that does this faster but I am not sure which one.
谢谢!
出于什么价值,ngrams作为 nltk.util.ngrams 和 ngramlist
只是由所有找到的ngram组成的列表.
for what it's worth the ngrams are producted as joined output of nltk.util.ngrams and ngramlist
is just a list made from set of all found ngrams.
Edit2:
这里是可复制的代码来测试频率列表行(其余的代码并不是我真正关心的)
Here is reproducible code to test the freqlist line (the rest of the code is not really what I care about)
from nltk.util import ngrams
import wikipedia
import nltk
import pandas as pd
articles = ['New York City','Moscow','Beijing']
tokenizer = nltk.tokenize.TreebankWordTokenizer()
data={'article':[],'treebank_tokenizer':[]}
for article in articles:
data['article' ].append(wikipedia.page(article).content)
data['treebank_tokenizer'].append(tokenizer.tokenize(data['article'][-1]))
df=pd.DataFrame(data)
df['ngrams-3']=df['treebank_tokenizer'].map(
lambda x: [' '.join(t) for t in ngrams(x,3)])
ngramlist = list(set([trigram for sublist in df['ngrams-3'].tolist() for trigram in sublist]))
df['freqlist']=df['ngrams-3'].map(lambda ngrams_: [sum(int(ngram==ngram_condidate) for ngram_condidate in ngrams_)/len(ngrams_) for ngram in ngramlist])
推荐答案
You can probably optimize this a bit by pre-computing some quantities and using a Counter
. This will be especially useful if most of the elements in ngramlist
are contained in ngrams
.
freqlist = [
sum(int(ngram == ngram_candidate)
for ngram_candidate in ngrams) / len(ngrams)
for ngram in ngramlist
]
当然,您每次检查 ngram
时都不需要遍历 ngrams
.一遍 ngrams
将使此算法为 O(n)
,而不是 O(n 2 )
现在.请记住,较短的代码不一定是更好或更有效的代码:
You certainly don't need to iterate over ngrams
every single time you check an ngram
. One pass over ngrams
will make this algorighm O(n)
instead of the O(n2)
one you have now. Remember, shorter code is not necessarily better or more efficient code:
from collections import Counter
...
counter = Counter(ngrams)
size = len(ngrams)
freqlist = [counter.get(ngram, 0) / size for ngram in ngramlist]
要正确使用此功能,您必须编写一个 def
函数而不是一个 lambda
:
To use this function properly, you would have to write a def
function instead of a lambda
:
def count_ngrams(ngrams):
counter = Counter(ngrams)
size = len(ngrams)
freqlist = [counter.get(ngram, 0) / size for ngram in ngramlist]
return freqlist
df['freqlist'] = df['ngrams-3'].map(count_ngrams)
这篇关于标记化文本中ngram(字符串)的出现频率的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!