Solr 搜索主题标签或提及 [英] Solr search for hashtag or mentions

问题描述

我们使用 solr 3.5 版来搜索推文，我使用 WordDelimiterFactory 和以下设置，以便能够搜索 @username 或 #标签:

We are using solr version 3.5 to search though Tweets, I am using WordDelimiterFactory with the following setting, to be able to search for @username or #hashtags:

<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="0" splitOnNumerics="0" preserveOriginal="1" handleAsChar="@#"/>

我看到了以下补丁，但这似乎并没有像我预期的那样工作，我是否遗漏了什么?

I saw the following patch but this doesn’t seem to be working as I expected, am I missing something?

https://issues.apache.org/jira/browse/SOLR-2059

但是搜索 @username 也只返回用户名的结果，或者 #hashtag 只返回 hastag 的结果.我怎样才能做到这一点?

But searching for @username is also returning results for just username or #hashtag is just returning result for hastag. How can I achieve this?

整个字段类型:

<fieldType name="textnostem" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true">
  <analyzer type="index">
    <charFilter class="solr.HTMLStripCharFilterFactory"/>
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords.txt"
            enablePositionIncrements="true"
            />
    <filter class="solr.WordDelimiterFilterFactory" 
            generateWordParts="1" 
            generateNumberParts="1" 
            catenateWords="1" 
            catenateNumbers="1" 
            catenateAll="0" 
            splitOnCaseChange="0" 
            splitOnNumerics="0"
            preserveOriginal="1"
            />
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords.txt"
            enablePositionIncrements="true"
            />
    <filter class="solr.WordDelimiterFilterFactory" 
            generateWordParts="1" 
            generateNumberParts="1" 
            catenateWords="1" 
            catenateNumbers="1" 
            catenateAll="0" 
            splitOnCaseChange="0" 
            splitOnNumerics="0"
            preserveOriginal="1"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>

<fieldType name="textnostem" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true">
  <analyzer type="index">
    <charFilter class="solr.HTMLStripCharFilterFactory"/>
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords.txt"
            enablePositionIncrements="true"
            />
    <filter class="solr.WordDelimiterFilterFactory" 
            generateWordParts="1" 
            generateNumberParts="1" 
            catenateWords="1" 
            catenateNumbers="1" 
            catenateAll="0" 
            splitOnCaseChange="0" 
            splitOnNumerics="0"
            preserveOriginal="1"
            />
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords.txt"
            enablePositionIncrements="true"
            />
    <filter class="solr.WordDelimiterFilterFactory" 
            generateWordParts="1" 
            generateNumberParts="1" 
            catenateWords="1" 
            catenateNumbers="1" 
            catenateAll="0" 
            splitOnCaseChange="0" 
            splitOnNumerics="0"
            preserveOriginal="1"/>      
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>

<fieldType name="textnostem" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true">
  <analyzer type="index">
    <charFilter class="solr.HTMLStripCharFilterFactory"/>
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords.txt"
            enablePositionIncrements="true"
            />
    <filter class="solr.WordDelimiterFilterFactory" 
            generateWordParts="1" 
            generateNumberParts="1" 
            catenateWords="1" 
            catenateNumbers="1" 
            catenateAll="0" 
            splitOnCaseChange="0" 
            splitOnNumerics="0"
            preserveOriginal="1"
            handleAsChar="@#"
            />
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords.txt"
            enablePositionIncrements="true"
            />
    <filter class="solr.WordDelimiterFilterFactory" 
            generateWordParts="1" 
            generateNumberParts="1" 
            catenateWords="1" 
            catenateNumbers="1" 
            catenateAll="0" 
            splitOnCaseChange="0" 
            splitOnNumerics="0"
            preserveOriginal="1"
            handleAsChar="@#"
            />      
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>

推荐答案

好的，通读你提到的 SOLR-2059 补丁，看起来他们已经替换了 WordDelimiterFactory 上的 handleAsChar 属性使用 types 属性.以下是来自分析器、标记器和标记过滤器 Solr Wiki 页面的该属性的规范:

OK, so reading through the SOLR-2059 patch that you mentioned, it looks like they have replaced the handleAsChar attribute on the WordDelimiterFactory with the types attribute. Here is the specification for that attribute from the Analyzers, Tokenizers and Token Filters Solr Wiki page:

types="wdfftypes.txt" 允许对此过滤器进行自定义标记化.该文件应该存在于 solr/conf 目录中，并且条目的形式(不带引号)% => ALPHA"或u002C => DIGIT".允许的类型有:LOWER、UPPER、ALPHA、DIGIT、ALPHANUM、SUBWORD_DELIM.

types="wdfftypes.txt" allows customized tokenization for this filter. The file should exist in the solr/conf directory, and entries are of the form (without quotes) "% => ALPHA" or "u002C => DIGIT". Allowable types are: LOWER, UPPER, ALPHA, DIGIT, ALPHANUM, SUBWORD_DELIM.

因此，如果我们采用此文档，再加上 SOLR-2059 中的文件示例，我会推荐以下内容:

So then if we take this documentation, plus the example of the file from SOLR-2059, I would recommend the following:

<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="0" splitOnNumerics="0"preserveOriginal="1" types="twittertypes.txt"/>

然后按如下方式定义 twittertypes.txt 文件，并将其放置在与您的 Solr 实例中的 schema.xml 文件相同的文件夹中(可能是 conf 文件夹).

Then define the twittertypes.txt file as follows and place it in the same folder as your schema.xml file in your Solr instance (probably the conf folder).

 # A customized type mapping for WordDelimiterFilterFactory
 # the allowable types are: LOWER, UPPER, ALPHA, DIGIT, ALPHANUM, SUBWORD_DELIM
 #    
 # the default for any character without a mapping is always computed from 
 # Unicode character properties

 # Map the $, %, '.', and ',' characters to DIGIT 
 # This might be useful for financial data.
 @ => ALPHA
 u0023 => ALPHA

请注意，哈希符号需要使用 Unicode 字符 (UTF-8)，因为它在文本文件中被视为注释.

Notice that you need to use the Unicode character (UTF-8) for the hash symbol, since it is treated as comment in the text file.

根据所有文档，这应该可以解决您的问题并将 # 和 @ 符号视为字母字符，这将提供您正在寻找的行为.

According to all of the documentation, this should fix your issue and treat the # and @ symbols as alpha characters, which will provide the behavior you are looking for.

这篇关于Solr 搜索主题标签或提及的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！