Elasticsearch - 如果字段具有相同的名称但是具有多个映射，该怎么做 [英] Elasticsearch - what to do if fields have the same name but multiple mapping

问题描述

我使用Elasticsearch存储从系统外部的多个源发送的数据，即我不控制传入的数据 - 我只是接收json文档并存储。我的中间没有logstash的过滤器，只有ES和Kibana。每个数据源发送自己的数据类型，并且它们都存储在相同的索引（每个租户）中，但是以不同的类型存储。然而，由于我无法控制发送给我的数据，所以可以接收具有相同名称和不同结构的字段的不同类型的文档。

例如，假设我有type1和type2与字段FLD，它是一个对象在这两种情况下，但该对象的结构是不一样的。特别是FLD.name是type1中的一个字符串字段，但是type2中的一个对象。在这种情况下，当type1数据到达时，它成功存储，但是当type2数据到达时，它被拒绝：

I use Elasticsearch for storing data sent from multiple sources outside of my system, i.e. I'm not controlling the incoming data - I just receive json document and store it. I have no logstash with its filters in the middle, only ES and Kibana. Each data source sent its own data type and all of them are stored in the same index (per tenant) but in different types. However since I cannot control the data that is sent to me, it is possible to receive documents of different types with the field having the same name and different structure.
For example, assume that I have type1 and type2 with field FLD, which is an object in both cases but the structure of this object is not the same. Specifically FLD.name is a string field in type1 but an object in type2. And in this case, when type1 data arrives it is stored successfully but when type2 data arrives, it is rejected:

无法将映射置于index [[myindex]]，type [type2]

java.lang.IllegalArgumentException：[FLD]的Mapper与其他类型的现有映射冲突[无法合并非对象映射[FLD.name]使用对象映射[FLD.name]]

failed to put mappings on indices [[myindex]], type [type2]
java.lang.IllegalArgumentException: Mapper for [FLD] conflicts with existing mapping in other types[Can't merge a non object mapping [FLD.name] with an object mapping [FLD.name]]

ES文档清楚地声明在不同映射中的相同索引中具有相同名称的字段类型在内部映射到相同的字段，并且必须具有相同的映射（见这里）。

ES documentation clearly declare that fields with the same name in the same index in different mapping types mapped to the same field internally and must have the same mapping (see here).

我的问题在这种情况下可以做什么？我宁愿将所有类型保留在同一个索引中。可以为字段名称添加一个unique-per-type后缀吗？任何其他解决方案我是Elasticsearch的新手，所以也许我错过了一些简单的事情...提前感谢。

My question is what can I do in this case? I'd prefer to keep all the types in the same index. Is it possible to add a unique-per-type suffix to field names or something like this? Any other solution? I'm a newbie in Elasticsearch so maybe I'm missing something simple... Thanks in advance.

推荐答案

没有在索引索引之前，无需进行预处理即可实现索引任意JSON的方式 - 甚至

There is no way to do index arbitrary JSON without pre-processing before it's indexed - not even Dynamic templates are flexible enough.

您可以将嵌套对象平铺成键值对，并使用嵌套数据类型，多字段和 ignore_malformed 来索引任意的JSON（即使是类型冲突），如上所述 here 。不幸的是，如果您尝试将字符串匹配为 kv_pairs.value.long ，那么Elasticsearch仍然可以在查询时抛出异常，因此您可以选择适当的字段基于价值的格式。

You can flatten nested objects into key-value pairs and use a Nested datatype, Multi-fields, and ignore_malformed to index arbitrary JSON (even with type conflicts) as described here. Unfortunately, Elasticsearch can still throw an exception at query time if you try to, for example, match a string to kv_pairs.value.long, so you'll have choose appropriate fields based on format of the value.

这篇关于Elasticsearch - 如果字段具有相同的名称但是具有多个映射，该怎么做的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！