如何将带有聚合的 Elasticsearch JSON 字符串响应转换为 Elasticsearch SearchResponse 对象 [英] How do you convert an Elasticsearch JSON String Response, with an Aggregation, to an Elasticsearch SearchResponse Object

问题描述

我想将一个 json 字符串序列化为一个 Elasticsearch SearchResponse 对象.如果 json 字符串不包含聚合，它就可以正常工作.

I want to serialize a json string to an Elasticsearch SearchResponse object. It works fine if the json string doesn't contains an aggregation.

如果 json 字符串包含聚合，则 XContentParser 会抛出一个ParsingException[无法解析以 [target_field] 为键的聚合例外.

If the json string contains an aggregation the XContentParser throws an ParsingException[Could not parse aggregation keyed as [target_field] exception.

我用来将 json 字符串序列化为 Elasticsearch SearchResponse 对象的代码:

The code I use to serialize the json string to an Elasticsearch SearchResponse object:

    Settings settings = Settings.builder().build();
    SearchModule searchModule = new SearchModule(settings, false, new ArrayList<>());

    NamedXContentRegistry xContentRegistry = new NamedXContentRegistry(searchModule.getNamedXContents());

    JsonXContentParser xContentParser = new JsonXContentParser(xContentRegistry,
            new JsonFactory().createParser(json));
    SearchResponse response = SearchResponse.fromXContent(xContentParser);

似乎我必须向 NamedXContentRegistry 注册聚合，但我不知道如何进行.

It seems that I have to register aggregations to the NamedXContentRegistry but i don't know how to.

推荐答案

背景:
我正在根据我为编写 Java 单元测试而创建 SearchResponse 对象的经验编写此答案.目标是从 Elasticsearch 查询中获取任何 JSON 响应对象，将其编组到 SearchResponse 对象中，然后对创建可消费输出的业务逻辑进行单元测试.

Background:
I'm writing this answer from my experience of creating a SearchResponse object for the purpose of writing a Java Unit Test. The goal was to take any JSON response object from an Elasticsearch query, marshall that into a SearchResponse object, and Unit Test the business logic of creating a consumable output.

我们正在使用 Elasticsearch 6.7，即高级别的客户端，并使用 Elastic 的 POJO 解析 SearchResponse(而不是仅执行 .toString() 并使用 GSON 或 Jackson 对其进行操作).

We're using Elasticsearch 6.7, the high-level rest client, and parsing the SearchResponse using Elastic's POJOs (vs just doing a .toString() and manipulating it with GSON or Jackson).

解决方案说明:
Elasticsearch 的高级休息客户端一般会解析来自低级休息客户端的结果.SearchRequest 的响应 JSON 被转换为 RestHighLevelClient 搜索方法.此方法调用 performRequestAndParseEntity 在第 1401 行，它接受 entityParser 作为 CheckedFunction.最后，我们可以看到在调用 entityParser 在第 1401 行，它调用 parseEntity 方法在第 1714 行它确定实体的 XContentType 并最终执行解析.值得注意的是，当解析器在第 1726 行创建时，registry 被传递到解析器.这个注册表包含响应字段可能是的所有可能的 XContent 值.注册表是在 第 288 行.类型的完整列表，包括聚合类型，列在 第 1748 行.

Explanation of the solution:
Elasticsearch's high-level rest client generically parses results from the low-level rest client. The SearchRequest's response JSON is converted into a SearchResponse Object in the RestHighLevelClient on line 129 in the search method. This method calls performRequestAndParseEntity on line 1401, which accepts an entityParser as a CheckedFunction<XContentParser, Resp, IOException>. Finally, we can see that when invoking the entityParser on line 1401, it calls the parseEntity method on line 1714 which determines the XContentType for the entity and ultimately performs the parse. Notably, when the parser is created on line 1726 a registry is passed into the parser. This registry contains all the possible XContent values a response field may be. The registry is created when the RestHighLevelClient is constructed on line 288. The full list of types, including Aggregation types, is listed on line 1748.

解决方案:
阅读 Elasticsearch 对此的讨论后，如果您想将来自 Elastic 的 JSON Response 注入 SearchResponse 对象，则需要创建一个 NamedXContentRegistry 并列出XContents 测试你必须重新创建解析.一个帮助方法来做到这一点，来源于 Elastic 的讨论:

Onto the solution:
After reading the Elasticsearch discussion on this, it would appear that if you want to inject a JSON Response from Elastic into the SearchResponse object, it is necessary to create a NamedXContentRegistry and list of XContents testing you have to re-create the parsing. A helper method to do that, sourced from Elastic's discussion:

public static List<NamedXContentRegistry.Entry> getDefaultNamedXContents() {
    Map<String, ContextParser<Object, ? extends Aggregation>> map = new HashMap<>();
    map.put(TopHitsAggregationBuilder.NAME, (p, c) -> ParsedTopHits.fromXContent(p, (String) c));
    map.put(StringTerms.NAME, (p, c) -> ParsedStringTerms.fromXContent(p, (String) c));
    List<NamedXContentRegistry.Entry> entries = map.entrySet().stream()
            .map(entry -> new NamedXContentRegistry.Entry(Aggregation.class, new ParseField(entry.getKey()), entry.getValue()))
            .collect(Collectors.toList());
  return entries;
}

上述代码中的地图需要所有您的测试所需的聚合.有两个以上，为了简洁起见，这里有两个.

The map in the above code needs to have ALL of the Aggregations that's necessary for your test. There are more than two, two are here for brevity.

使用这个助手 getNamedXContents() 方法，您现在可以使用以下方法获取 JSON 字符串并将其注入 SearchResponse.同样来自 Elastic 的讨论:

Using this helper getNamedXContents() method, you can now use the following method to take a JSON String and inject it into the SearchResponse. Also sourced from Elastic's Discussion:

public static SearchResponse getSearchResponseFromJson(String jsonResponse){
    try {
        NamedXContentRegistry registry = new NamedXContentRegistry(getDefaultNamedXContents());
        XContentParser parser = JsonXContent.jsonXContent.createParser(registry, jsonResponse);
        return SearchResponse.fromXContent(parser);
    } catch (IOException e) {
        System.out.println("exception " + e);
    }catch (Exception e){
        System.out.println("exception " + e);
    }
    return new SearchResponse();
}

应用带有聚合结果的解决方案:
Elasticsearch 需要一个提示来知道将其解析为哪种类型的聚合.将 ?typed_keys 添加到查询时，该提示由 elastic 提供.关于 聚合类型的 Elasticsearch 文档中显示了一个示例提示.

Applying the solution with an Aggregation result:
Elasticsearch needs a hint to know what type of aggregation to parse this as. The hint is provided by elastic when adding ?typed_keys to the query. An example is shown in the Elasticsearch documentation on Aggregation Type Hints.

要将 JSON 字符串注入 SearchResponse 对象，必须 (1) 使用上述方法和 (2) 注入带有类型提示的字符串.

To inject the JSON String into a SearchResponse object, one must (1) Use the methods above and (2) Inject a string with type hints in it.

主要来源:

1. https://discuss.elastic.co/t/elasticsearch-json-response-to-searchresponse-object/124394/6
2. https://github.com/elastic/elasticsearch/blob/master/client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLevelClient.java
3. https://github.com/elastic/elasticsearch/blob/master/test/framework/src/main/java/org/elasticsearch/test/InternalAggregationTestCase.java
4. https://www.elastic.co/guide/en/elasticsearch/reference/current/returning-aggregation-type.html

注意:大约 2015 年有很多文章说这是不可能的.这显然是错误的.

Note: There are a lot of articles from circa-2015 that say this is impossible. That is obviously incorrect.

这篇关于如何将带有聚合的 Elasticsearch JSON 字符串响应转换为 Elasticsearch SearchResponse 对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！