我如何履行大量承诺？ [英] how do I perform a large batch of promises?

问题描述

我计划在firebase上运行大量查询，这些查询可能会增长到几十万甚至数百万。我一直在使用 Promise.all（）来解决大多数查询，但是随着请求的增长 Promise.all（）似乎只是停止以随机数运行。我已经研究过使用 Promise.map（），但是我不确定并发是否可以解决问题。谢谢您的帮助。

I am planning on running a large number of queries on firebase which could grow to the order of a few hundred thousand to even millions. I've been using Promise.all() to resolve most of my queries but as the requests grow Promise.all() seems to just stop running at a random number.Ive looked into using Promise.map() but Im not sure if the concurrency will solve the problem. Thank you for your help.

下面是一个简化的示例，您可以看到它似乎只是超时而不会引发任何错误：

Below is a simplified example as you can see it appears to just time out without throwing any error:

var promises = [];
var i = 0;
for(var x = 0; x<1000000; x++){
 const promise = new Promise((resolve, reject) => {
  setTimeout(() => {
    i += 1;
    resolve(i);
  }, 10);
});
  promises.push(promise);
}


Promise.all(promises).then((value) => {
  console.log(value)
}).catch((error) => {
  console.log("Error making js node work:" + error);
})

推荐答案

当我需要做这样的事情时，我通常将查询分为几批。批处理是一对一运行的，但是每个批处理中的查询是并行运行的。

When I need to do something like this, I usually divide the queries into batches. The batches run one-by-one, but the queries in each batch run in parallel. Here's what that might look like.

const _ = require('lodash');

async function runAllQueries(queries) {
  const batches = _.chunk(queries, BATCH_SIZE);
  const results = [];
  while (batches.length) {
    const batch = batches.shift();
    const result = await Promises.all(batch.map(runQuery));
    results.push(result)
  }
  return _.flatten(results);
}

您在此处看到的内容类似于地图缩减。也就是说，如果您在单个节点（例如，单个进程或虚拟机）中运行大量查询，则可以考虑将查询分布在多个节点上。如果查询的数量很大，并且查询的处理顺序并不重要，那么这很容易。您还应该确保下游系统（即您要查询的系统）可以处理您向其施加的负载。

What you see here is similar to a map-reduce. That said, if you're running a large number of queries in a single node (e.g., a single process or virtual machine), you might consider distributing the queries across multiple nodes. If the number of queries is very large and the order in which the queries are processed is not important, this is probably a no-brainer. You should also be sure that the downstream system (i.e., the one you're querying) can handle the load you throw at it.

这篇关于我如何履行大量承诺？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！