检测重复的英文名称 [英] Detect duplicate English names

问题描述

我试图找到一个例子，演示一个Lucene或其他一些索引，可以检查英语第一&可能重复的姓氏组合。重复的检查需要能够考虑常见的昵称，即Bob for Robert和Bill for William以及拼写错误。有谁知道一个例子？

I'm attempting to find an example that demonstrates a Lucene or some other kind of index that can check an English first & last name combination for possible duplicates. The duplicate check needs to be able to take into account common nicknames, i.e. Bob for Robert and Bill for William, as well as spelling mistakes. Does anyone know of an example?

我计划在用户注册期间执行重复搜索。新的用户记录需要根据存储用户名的数据库表创建的索引进行检查。

I plan to perform the duplicates search during user registration. The new user record needs to be checked against an index that has been built from the database table that stores the user names.

推荐答案

我将在索引时在firstName上使用一个SynonymFilter，以便您具有所有可能的组合（Bob - > Robert，Robert - > Bob等）。索引您现有的用户。

I would use a SynonymFilter on the firstName while indexing, so that you have all possible combinations (Bob -> Robert, Robert -> Bob, etc...). Index the existing users you have.

然后使用QueryParser（分析器中没有SynonymFilter）来询问一些模糊查询。

Then use the QueryParser (without the SynonymFilter in the analyzer) to ask some fuzzy queries.

这是我提出的代码：

public class NameDuplicateTests {
    private Analyzer analyzer;
    private IndexSearcher searcher;
    private IndexReader reader;
    private QueryParser qp;

    private final static Multimap<String, String> firstNameSynonyms;
    static {
        firstNameSynonyms = HashMultimap.create();
        List<String> robertSynonyms = ImmutableList.of("Bob", "Bobby", "Robert");
        for (String name: robertSynonyms) {
            firstNameSynonyms.putAll(name, robertSynonyms);
        }
        List<String> willSynonyms = ImmutableList.of("William", "Will", "Bill", "Billy");
        for (String name: willSynonyms) {
            firstNameSynonyms.putAll(name, willSynonyms);
        }
    }

    public static Analyzer createAnalyzer() {
        return new Analyzer() {
            @Override
            public TokenStream tokenStream(String fieldName, Reader reader) {
                TokenStream tokenizer = new WhitespaceTokenizer(reader);
                if (fieldName.equals("firstName")) {
                    tokenizer = new SynonymFilter(tokenizer, new SynonymEngine() {
                        @Override
                        public String[] getSynonyms(String s) throws IOException {
                            return firstNameSynonyms.get(s).toArray(new String[0]);
                        }
                    });
                }
                return tokenizer;
            }
        };
    }


    @Before
    public void setUp() throws Exception {
        Directory dir = new RAMDirectory();
        analyzer = createAnalyzer();

        IndexWriter writer = new IndexWriter(dir, analyzer, IndexWriter.MaxFieldLength.UNLIMITED);
        ImmutableList<String> firstNames = ImmutableList.of("William", "Robert", "Bobby", "Will", "Anton");
        ImmutableList<String> lastNames = ImmutableList.of("Robert", "Williams", "Mayor", "Bob", "FunkyMother");

        for (int id = 0; id < firstNames.size(); id++) {
            Document doc = new Document();
            doc.add(new Field("id", String.valueOf(id), Field.Store.YES, Field.Index.NOT_ANALYZED));
            doc.add(new Field("firstName", firstNames.get(id), Field.Store.YES, Field.Index.ANALYZED));
            doc.add(new Field("lastName", lastNames.get(id), Field.Store.YES, Field.Index.NOT_ANALYZED));
            writer.addDocument(doc);
        }
        writer.close();

        qp = new QueryParser(Version.LUCENE_30, "firstName", new WhitespaceAnalyzer());
        searcher = new IndexSearcher(dir);
        reader = searcher.getIndexReader();
    }

    @After
    public void tearDown() throws Exception {
        searcher.close();
    }

    @Test
    public void testNameFilter() throws Exception {
        search("+firstName:Bob +lastName:Williams");
        search("+firstName:Bob +lastName:Wolliam~");
    }

    private void search(String query) throws ParseException, IOException {
        Query q = qp.parse(query);
        System.out.println(q);
        TopDocs res = searcher.search(q, 3);
        for (ScoreDoc sd: res.scoreDocs) {
            Document doc = reader.document(sd.doc);
            System.out.println("Found " + doc.get("firstName") + " " + doc.get("lastName"));
        }
    }
}

其中： p>

Which results in:

+firstName:Bob +lastName:Williams
Found Robert Williams
+firstName:Bob +lastName:wolliam~0.5
Found Robert Williams

希望有帮助

这篇关于检测重复的英文名称的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！