Lucene/Solr 如何在多字段/分面搜索中实现高性能? [英] How does Lucene/Solr achieve high performance in multi-field / faceted search?

问题描述

上下文

这是一个主要关于 Lucene(或可能是 Solr)内部的问题.主要主题是分面搜索，其中可以沿着对象的多个独立维度(方面)进行搜索(例如汽车的大小、速度、价格).

This is a question mainly about Lucene (or possibly Solr) internals. The main topic is faceted search, in which search can happen along multiple independent dimensions (facets) of objects (for example size, speed, price of a car).

当使用关系数据库实现时，对于大量构面，多字段索引没有用，因为可以按任何顺序搜索构面，因此使用特定有序多索引的机会很小，并创建所有可能的排序指数难以忍受.

When implemented with relational database, for a large number of facets multi-field indices are not useful, since facets can be searched in any order, so a specific ordered multi-index is used with low chance, and creating all possible orderings of indices is unbearable.

Solr 被宣传为可以很好地应对分面搜索任务，如果我认为正确的话，它必须与 Lucene 相关联(据说)在多字段查询(文档的字段与对象的方面相关)上表现良好.

Solr is advertised to cope well with the faceted search task, which if I think correctly has to be connected with Lucene (supposedly) performing well on multi-field queries (where fields of a document relate to facets of an object).

问题

Lucene的倒排索引可以存储在关系型数据库中，自然取匹配文档的交集也可以通过RDBMS使用单字段索引轻松实现.

The inverted index of Lucene can be stored in a relational database, and naturally taking the intersections of the matching documents can also be trivially achieved with RDBMS using single-field indices.

因此，Lucene 应该有一些用于多字段查询的先进技术，而不仅仅是基于倒排索引获取匹配文档的交集.

Therefore, Lucene supposedly has some advanced technique for multi-field queries other than just taking the intersection of matching documents based on the inverted index.

所以问题是，这种技术/技巧是什么?更广泛地说:为什么 Lucene/Solr 在理论上可以实现比 RDBMS 更好的分面搜索性能(如果可以的话)?

So the question is, what is this technique/trick? More broadly: Why can Lucene/Solr achieve better faceted search performance theoretically than RDBMS could (if so)?

注意:我的第一个猜测是 Lucene 会使用一些空间分区方法来分割从文档字段构建的向量空间作为维度，但据我了解，Lucene 并不是纯粹基于向量空间的.

推荐答案

分面

分面有两种答案，因为分面有两种类型.我不确定其中任何一个都比 RDBMS 快.

There are two answers for faceting, because there are two types of faceting. I'm not certain that either of these are faster than an RDBMS.

1. 枚举分面.查询的结果是一个位向量，如果第 i 个文档是匹配的，则第 i 个位为 1.刻面也是位向量，因此交集只是按位与.我不认为这是一种新颖的方法，而且大多数 RDBMS 可能都支持它.

2. 字段缓存.这只是一个正常(非反转)索引.此处运行的 SQL 样式查询如下:

1. Enum faceting. Results of a query are a bit vector where the ith bit is 1 if the ith document was a match. The facet is also a bit vector, so intersection is just a bitwise AND. I don't think this is a novel approach, and most RDBMS's probably support it.

2. Field Cache. This is just a normal (non-inverted) index. The SQL-style query that is run here is like:

从 field_cache 中选择 facet，count(*)query_results 中的 docId 在哪里按方面分组

select facet, count(*) from field_cache where docId in query_results group by facet

再说一次，我不认为这是普通 RDBMS 无法做到的.索引是一个跳过列表，以 docId 为键.

Again, I don't think this is anything that a normal RDBMS couldn't do. The index is a skip list, with the docId as the key.

多词搜索

这就是 Lucene 的亮点.为什么 Lucene 的方法这么好就不在这里发了，不过我可以推荐 这篇文章关于 Lucene 性能，或其中链接的论文.

This is where Lucene shines. Why Lucene's approach is so good is too long to post here, but I can recommend this post on Lucene Performance, or the papers linked therein.

这篇关于Lucene/Solr 如何在多字段/分面搜索中实现高性能?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！