通过 spring-data 迭代 MongoDB 中的大型集合 [英] Iterate over large collection in MongoDB via spring-data

问题描述

朋友们！

我通过 spring-data 在 java 项目中使用 MongoDB.我使用 Repository 接口来访问集合中的数据.对于某些处理，我需要遍历集合的所有元素.我可以使用存储库的 fetchAll 方法，但它总是返回 ArrayList.

I am using MongoDB in java project via spring-data. I use Repository interfaces to access data in collections. For some processing I need to iterate over all elements of collection. I can use fetchAll method of repository, but it always return ArrayList.

但是，假设集合之一会很大 - 最多 100 万条记录，每个记录至少有几千字节.我想在这种情况下我不应该使用 fetchAll，但我找不到返回一些迭代器的方便方法(这可能允许部分获取集合)，也找不到带有回调的方便方法.

However, it is supposed that one of collections would be large - up to 1 million records several kilobytes each at least. I suppose I should not use fetchAll in such cases, but I could not find neither convenient methods returning some iterator (which may allow collection to be fetched partially), nor convenient methods with callbacks.

我见过只支持在页面中检索此类集合.我想知道这是否是处理此类集合的唯一方法?

I've seen only support for retrieving such collections in pages. I wonder whether it is the only way for working with such collections?

推荐答案

反应迟缓，但也许将来会帮助某人.Spring Data 不提供任何 API 来包装 Mongo DB Cursor 功能.它在 find 方法中使用它，但总是返回完整的对象列表.选项是直接使用 Mongo API 或使用 Spring Data Paging API，类似这样:

Late response, but maybe will help someone in the future. Spring data doesn't provide any API to wrap Mongo DB Cursor capabilities. It uses it within find methods, but always returns completed list of objects. Options are to use Mongo API directly or to use Spring Data Paging API, something like that:

        final int pageLimit = 300;
        int pageNumber = 0;
        Page<T> page = repository.findAll(new PageRequest(pageNumber, pageLimit));
        while (page.hasNextPage()) {
            processPageContent(page.getContent());
            page = repository.findAll(new PageRequest(++pageNumber, pageLimit));
        }
        // process last page
        processPageContent(page.getContent());

UPD (!) 这种方法对于大数据集不够(见@Shawn Bush 评论)请直接使用 Mongo API 处理这种情况.

UPD (!) This method is not sufficient for large sets of data (see @Shawn Bush comments) Please use Mongo API directly for such cases.

这篇关于通过 spring-data 迭代 MongoDB 中的大型集合的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！