如何在Solr中使用n元语法近似匹配? [英] How to use n-grams approximate matching with Solr?

问题描述

我们有一个电影和电视剧数据库，由于数据来自可靠性各不相同的许多来源，我们希望能够对剧集的标题进行模糊字符串匹配.我们在应用程序中使用Solr进行搜索，但是默认的匹配机制在单词级别上运行，这对于诸如标题之类的短字符串还不够好

We have a database of movies and series, and as the data comes from many sources of varying reliability, we'd like to be able to do fuzzy string matching on the titles of episodes. We are using Solr for search in our application, but the default matching mechanisms operate on word levels, which is not good enough for short strings, like titles

我过去使用n-gram近似匹配，很高兴发现Lucene(和Solr)开箱即用.不幸的是，我无法正确配置它.

I had used n-grams approximate matching in the past, and I was very happy to find that Lucene (and Solr) supports something this out of the box. Unfortunately, I haven't been able to configure it correctly.

我假设为此需要特殊的字段类型，因此我添加了 我的schema.xml的以下字段类型:

I assumed that I need a special field type for this, so I added the following field-type to my schema.xml:

<fieldType 
   name="trigrams" 
   stored="true" 
   class="solr.StrField"> 
 <analyzer type="index"> 
   <tokenizer 
       class="solr.analysis.NGramTokenizerFactory" 
       minGramSize="3" 
       maxGramSize="5" 
       /> 
   <filter class="solr.LowerCaseFilterFactory"/> 
 </analyzer> 
</fieldType> 

，并将架构中的相应字段更改为:

and changed the appropriate field in the schema to:

<field name="title" type="trigrams" 
    indexed="true" stored="true" multiValued="false" /> 

但是，这不符合我的预期.查询分析看起来 正确，但是我没有得到任何结果，这使我相信 在索引时间发生了一些事情(即标题被索引为 默认字符串字段而不是Trigram字段).

However, this is not working as I expected. The query analysis looks correctly, but I don't get any results, which makes me believe that something happens at index time (ie. the title is indexed like a default string field instead of trigram field).

我正在尝试的查询类似

title:"guy walks into a psychiatrist office"

(带有一两个错字)，并且应该与"Guy走进精神病医生办公室"相匹配.

(with a typo or two) and it should match "Guy Walks into a Psychiatrist Office".

(我不确定查询是否正确.)

(I am not really sure if the query is correct.)

此外，我希望能够做更多的事情.我想 小写字符串，删除所有标点符号和空格，删除 英文停用词和THEN会将字符串更改为三字组.然而， 仅在字符串被标记化之后才应用过滤器...

Moreover, I would like to be able to do something more in fact. I'd like to lowercace the string, remove all punctuation marks and spaces, remove English stopwords and THEN change the string into trigrams. However, the filters are applied only after the string has been tokenized...

预先感谢您的回答.

推荐答案

回答问题的最后一部分:solr还有一个ngram过滤器.因此，您不应该使用ngram标记器(例如，类似"WhitespaceTokenizer"的标记器)，应用所有ngram之前的过滤器，然后添加以下标记:

To answer to the last part of your question: solr has also an ngram filter. So you should not use the ngram tokenizer (but one like "WhitespaceTokenizer" for example), apply all pre-ngram filters and then add this one:

<filter class="solr.NGramFilterFactory" minGramSize="2" maxGramSize="3" />

这篇关于如何在Solr中使用n元语法近似匹配?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！