如何将具有聚合的Elasticsearch JSON字符串响应转换为Elasticsearch SearchResponse对象 [英] How do you convert an Elasticsearch JSON String Response, with an Aggregation, to an Elasticsearch SearchResponse Object

问题描述

我想将json字符串序列化为Elasticsearch SearchResponse对象.如果json字符串不包含聚合，则效果很好.

I want to serialize a json string to an Elasticsearch SearchResponse object. It works fine if the json string doesn't contains an aggregation.

如果json字符串包含一个聚合，则XContentParser会抛出一个 ParsingException [无法解析键为[target_field]的聚合

If the json string contains an aggregation the XContentParser throws an ParsingException[Could not parse aggregation keyed as [target_field] exception.

我用于将json字符串序列化为Elasticsearch SearchResponse对象的代码:

The code I use to serialize the json string to an Elasticsearch SearchResponse object:

    Settings settings = Settings.builder().build();
    SearchModule searchModule = new SearchModule(settings, false, new ArrayList<>());

    NamedXContentRegistry xContentRegistry = new NamedXContentRegistry(searchModule.getNamedXContents());

    JsonXContentParser xContentParser = new JsonXContentParser(xContentRegistry,
            new JsonFactory().createParser(json));
    SearchResponse response = SearchResponse.fromXContent(xContentParser);

似乎我必须将聚合注册到NamedXContentRegistry，但是我不知道该怎么做.

It seems that I have to register aggregations to the NamedXContentRegistry but i don't know how to.

推荐答案

背景:
我正在根据创建 SearchResponse 对象以编写 Java单元测试的经验来编写此答案.目标是从Elasticsearch查询中获取任何JSON响应对象，将其编组为 SearchResponse 对象，并对创建可消耗输出的业务逻辑进行单元测试.

Background:
I'm writing this answer from my experience of creating a SearchResponse object for the purpose of writing a Java Unit Test. The goal was to take any JSON response object from an Elasticsearch query, marshall that into a SearchResponse object, and Unit Test the business logic of creating a consumable output.

我们使用的是高级REST客户端Elasticsearch 6.7，并使用Elastic的POJO解析 SearchResponse (与仅执行.toString()并使用GSON或Jackson对其进行操作).

We're using Elasticsearch 6.7, the high-level rest client, and parsing the SearchResponse using Elastic's POJOs (vs just doing a .toString() and manipulating it with GSON or Jackson).

解决方案说明:
Elasticsearch的高级REST客户端通常解析低级REST客户端的结果. SearchRequest 的响应JSON在 performRequestAndParseEntity ，它接受entityParser作为CheckedFunction<XContentParser, Resp, IOException>.最后，我们可以看到在调用

Explanation of the solution:
Elasticsearch's high-level rest client generically parses results from the low-level rest client. The SearchRequest's response JSON is converted into a SearchResponse Object in the RestHighLevelClient on line 129 in the search method. This method calls performRequestAndParseEntity on line 1401, which accepts an entityParser as a CheckedFunction<XContentParser, Resp, IOException>. Finally, we can see that when invoking the entityParser on line 1401, it calls the parseEntity method on line 1714 which determines the XContentType for the entity and ultimately performs the parse. Notably, when the parser is created on line 1726 a registry is passed into the parser. This registry contains all the possible XContent values a response field may be. The registry is created when the RestHighLevelClient is constructed on line 288. The full list of types, including Aggregation types, is listed on line 1748.

解决方案:
阅读有关此内容的Elasticsearch讨论之后，似乎您想将来自Elastic的JSON响应注入到 SearchResponse 对象中，有必要创建一个 NamedXContentRegistry 并列出XContents测试中，您必须重新创建解析.一种帮助方法，来自Elastic的讨论:

Onto the solution:
After reading the Elasticsearch discussion on this, it would appear that if you want to inject a JSON Response from Elastic into the SearchResponse object, it is necessary to create a NamedXContentRegistry and list of XContents testing you have to re-create the parsing. A helper method to do that, sourced from Elastic's discussion:

public static List<NamedXContentRegistry.Entry> getDefaultNamedXContents() {
    Map<String, ContextParser<Object, ? extends Aggregation>> map = new HashMap<>();
    map.put(TopHitsAggregationBuilder.NAME, (p, c) -> ParsedTopHits.fromXContent(p, (String) c));
    map.put(StringTerms.NAME, (p, c) -> ParsedStringTerms.fromXContent(p, (String) c));
    List<NamedXContentRegistry.Entry> entries = map.entrySet().stream()
            .map(entry -> new NamedXContentRegistry.Entry(Aggregation.class, new ParseField(entry.getKey()), entry.getValue()))
            .collect(Collectors.toList());
  return entries;
}

上面代码中的地图需要具有测试所需的汇总的 ALL .有两个以上，为简洁起见，这里有两个.

The map in the above code needs to have ALL of the Aggregations that's necessary for your test. There are more than two, two are here for brevity.

使用此助手getNamedXContents()方法，您现在可以使用以下方法获取JSON字符串并将其注入SearchResponse中. 也源自Elastic的讨论:

Using this helper getNamedXContents() method, you can now use the following method to take a JSON String and inject it into the SearchResponse. Also sourced from Elastic's Discussion:

public static SearchResponse getSearchResponseFromJson(String jsonResponse){
    try {
        NamedXContentRegistry registry = new NamedXContentRegistry(getDefaultNamedXContents());
        XContentParser parser = JsonXContent.jsonXContent.createParser(registry, jsonResponse);
        return SearchResponse.fromXContent(parser);
    } catch (IOException e) {
        System.out.println("exception " + e);
    }catch (Exception e){
        System.out.println("exception " + e);
    }
    return new SearchResponse();
}

应用具有聚合结果的解决方案:
Elasticsearch需要一个提示来了解将其解析为哪种聚合类型.当向查询中添加?typed_keys 时，elastic会提供提示. Elasticsearch文档中的聚合键入提示.

Applying the solution with an Aggregation result:
Elasticsearch needs a hint to know what type of aggregation to parse this as. The hint is provided by elastic when adding ?typed_keys to the query. An example is shown in the Elasticsearch documentation on Aggregation Type Hints.

要将JSON字符串注入到 SearchResponse 对象中，必须(1)使用上述方法，以及(2)注入其中带有类型提示的字符串.

To inject the JSON String into a SearchResponse object, one must (1) Use the methods above and (2) Inject a string with type hints in it.

主要来源:

1. https://discuss.elastic .co/t/elasticsearch-json-response-to-searchresponse-object/124394/6
2. https://github.com/elastic/elasticsearch/blob/master/client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLevelClient.java
3. https://github.com/elastic/elasticsearch/blob/master/test/framework/src/main/java/org/elasticsearch/test/InternalAggregationTestCase.java
4. https://www.elastic .co/guide/en/elasticsearch/reference/current/returning-aggregation-type.html

1. https://discuss.elastic.co/t/elasticsearch-json-response-to-searchresponse-object/124394/6
2. https://github.com/elastic/elasticsearch/blob/master/client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLevelClient.java
3. https://github.com/elastic/elasticsearch/blob/master/test/framework/src/main/java/org/elasticsearch/test/InternalAggregationTestCase.java
4. https://www.elastic.co/guide/en/elasticsearch/reference/current/returning-aggregation-type.html

注意:2015年左右有很多文章说这是不可能的.这显然是不正确的.

Note: There are a lot of articles from circa-2015 that say this is impossible. That is obviously incorrect.

这篇关于如何将具有聚合的Elasticsearch JSON字符串响应转换为Elasticsearch SearchResponse对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！