在提取管道中使用搜索模板 [英] Using a search template in an ingest pipeline

问题描述

ElasticSearch接收管道可以使用搜索模板作为其脚本吗？

Can an ElasticSearch ingest pipeline use a search template as its script?

具体来说，我想配置一个接收管道，以便每当特定类型的数据进来之后，我们在ElasticSearch中查询一些相关数据，然后根据结果在原始数据上填充一个或多个字段，然后再将其加载。

Specifically, I'd like to configure an ingest pipeline so that whenever data of a particular type come in, we query ElasticSearch for some related data, and populate a field (or fields) on the original data based on the results, before loading it in.

提取管道可以使用脚本（ https：// www .elastic.co / guide / zh-CN / elasticsearch / reference / master / script-processor.html ），并且该脚本可以包含搜索模板（> https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using。 html ），但我无法找到有关可能将两者结合的任何信息。

I see that ingest pipelines can use scripts (https://www.elastic.co/guide/en/elasticsearch/reference/master/script-processor.html), and that scripts can include search templates (https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html), but I haven't been able to find any information on potentially combining the two.

解决方案

实际上，您所要求的功能是通过新的丰富处理器，它提供了索引时JOIN功能。

Actually, the feature you're asking for is coming in 7.5 via the new enrich processor, which kind of provides index-time JOIN capability.

主要想法是设置丰富策略，它将把您的相关索引中的数据提供到一个新的丰富索引中，然后您可以使用丰富处理器，以使您的文档包含相关字段。

The main idea is to set up an enrich policy that will source data from your related indexes into a new "enrich index" and then you can leverage that "enrich index" in your ingest pipeline using an enrich processor in order to enrich your documents with related fields.

因此，无需赘述太多，以下是它的实际工作原理：

So, without going in too many details, here is how it works in practice:

1. 您有一个索引 A 与字段（ a ， b ， c ， d ），以用于丰富传入文档


2. 您可以基于索引 A定义扩展策略和 join字段 a


3. 您定义了具有丰富处理器的提取管道，该管道将尝试匹配传入文档的字段 z 与丰富索引的字段 Aa 相对


4. 如果找到匹配项，则您的传入文档将获得 b ， c 和 d 来自索引 A 。请注意，它还会获得匹配字段a，您可以根据需要使用删除处理器将其删除。

1. You have an index A with fields (a, b, c, d) that you'd like to use for enriching your incoming documents
2. You define an enrich policy based on that index A and the "join" field a
3. You define an ingest pipeline with an enrich processor that will try to match field z of the incoming document against field A.a of the enrich index
4. If a match is found, your incoming document will get fields b, c and d from the index A. Note that it will also get the match field a that you can remove using a remove processor if needed.

这几乎就是您所期望的。您可以在此处。一开始，它将适用于完全匹配（即 term 查询）和地理位置匹配（即 geo_shape 查询），但它们可能会添加新的匹配项（例如 range 匹配）。

That's pretty much what you expect. You can find a complete example here. At the beginning, it will work for exact matches (i.e. term query) and geo matches (i.e. geo_shape query), but they will probably add new kind of matches (like range matches) in the near future.

这篇关于在提取管道中使用搜索模板的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！