猫鼬光标:http集合中的批量请求 [英] Mongoose Cursor: http bulk request from collection

问题描述

我遇到一个与rxJS和大量集合中的大量HTTP请求(1M +文档)相关的问题

I have a problem which is relevant to rxJS and bulk HTTP requests from the huge collection (1M+ docs)

我有以下代码，逻辑很简单.我将集合中的所有文档推送到allplayers数组，并一次向API发出20个批量HTTP请求(猜测您了解其局限性)，因此，代码可以正常工作，但我想是时候从此重构它了:

I have the following code, with quite simple logic. I push all the docs from the collection to allplayers array and making bulk 20 HTTP requests to API at once (guess you understand why it's limited) So, the code works fine, but I guess it's time to refactor it from this:

        const cursor = players_db.find(query).lean().cursor();
        cursor.on('data', function(player) { allPlayers.push(player); });
        cursor.on('end', function() {
            logger.log('warng',`S,${allPlayers.length}`);
            from(allPlayers).pipe(
                mergeMap(player => getPlayer(player.name, player.realm),20),
            ).subscribe({
                next: player => console.log(`${player.name}@${player.realm}`),
                error: error => console.error(error),
                complete: () => console.timeEnd(`${updatePlayer.name}`),
            });
        });

就目前而言，我正在将find与cursor与(batchSize)配合使用，但是如果我理解此权限(通过.length)，并根据以下问题进行了解答:{

As for now, I'm using find with cursor with (batchSize), but if I understood this right (via .length), and according to this question: {mongoose cursor batchSize} batchSize is just a way of optimization and it's not return me array of X docs.

那我现在应该怎么做?我应该为rxJS选择什么运算符?

So what should I do now and what operator should I choose for rxJS?

例如，我可以形成具有必要长度(如20)的数组，并将其传输到rxJS，就像我以前使用的那样.但我想应该有另一种方法，可以在for promise loop

For example I could form arrays with necessary length (like 20) and transfer it to rxJS as I use it before. But I guess there should be another way, where I could use rxJS inside this for promise loop

    const players = await players_db.find(query).lean().cursor({batchSize: 10});
    for (let player = await players.next(); player != null; player = await players.next()) {
        //do something via RxJS inside for loop
    }

我也发现了这个问题{

Also I found this question {Best way to query all documents from a mongodb collection in a reactive way w/out flooding RAM} which also relevant to my problem and I understand the logic, but don't the syntax of it. I also know that cursor variable isn't a doc I cann't do anything useful with it. Or actually I could?

1. rxJS的bufferCount是一个非常有趣的运算符
2. https://gist.github.com/wellcaffeinated/f908094998edf54dc5840c8c3ad734d3 可能的解决方案?

1. rxJS's bufferCount is a quite interesting operator
2. https://gist.github.com/wellcaffeinated/f908094998edf54dc5840c8c3ad734d3 probable solution?

推荐答案

因此，最终我发现在这种情况下不需要rxJS.

So, in the end I found that rxJS isn't needed (but can be used) for this case.

解决方案非常简单，只需使用MongoCursor:

The solution was quite simple and using just MongoCursor:

async function BulkRequest (bulkSize = 10) {
    try {
       let BulkRequest_Array = [];
       const cursor = collection_db.find({}).lean().cursor({batchSize: bulkSize});
       cursor.on('data', async (doc) => {
           BulkRequest_Array.push(/*any function or axios instance*/)
            if (BulkRequest_Array.length >= bulkSize) {
                cursor.pause();
                console.time(`========================`);;
                await Promise.all(BulkRequest_Array);
                BulkRequest_Array.length = 0;
                cursor.resume();
                console.timeEnd(`========================`);
            }
       }
    } catch (e) {
       console.error(e)
    }
}

BulkRequest();

这篇关于猫鼬光标:http集合中的批量请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！