用于 Elasticsearch 扫描和滚动的 spring-batch ItemReader [英] spring-batch ItemReader for Elasticsearch scan and scroll

问题描述

在 spring-batch 中，是否支持 Elasticsearch ItemReader，使用扫描和滚动功能?我确实看到了 this 扩展，但是这是基于正常的弹簧数据搜索查询.有一个基于扫描和滚动功能的会很好，因为批处理作业主要需要处理大量数据.谢谢.

In spring-batch, is there any support for Elasticsearch ItemReader, using the scan and scroll functionality? I do see this extension, but that's based on the normal spring-data search queries. Would be nice to have one based on the scan and scroll feature, since the batch job mostly would need to process large batch of data. Thanks.

推荐答案

import java.util.Iterator;

import org.springframework.batch.item.data.AbstractPaginatedDataItemReader;
import org.springframework.data.elasticsearch.core.ElasticsearchOperations;
import org.springframework.data.elasticsearch.core.query.SearchQuery;

public class ElasticsearchItemReader<T> extends AbstractPaginatedDataItemReader<T> {
    private ElasticsearchOperations elasticsearchOperations;
    private final SearchQuery searchQuery;
    private String scrollId;
    private int scrollTimeinMillis = 60000;
    private Class<T> type;

    public ElasticsearchItemReader(
        final ElasticsearchOperations elasticsearchOperations,
        final SearchQuery searchQuery,
        final Class<T> type
    ) {
        this.elasticsearchOperations = elasticsearchOperations;
        this.searchQuery = searchQuery;
        this.type = type;
    }

    @Override
    protected void doOpen() throws Exception {
        scrollId = elasticsearchOperations.scan(searchQuery, scrollTimeinMillis, false);
    }

    @Override
    protected Iterator<T> doPageRead() {
        return elasticsearchOperations.scroll(scrollId, scrollTimeinMillis, type).iterator();
   }
}

这篇关于用于 Elasticsearch 扫描和滚动的 spring-batch ItemReader的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！