Promise.all消耗我所有的RAM [英] Promise.all consumes all my RAM
问题描述
我有一个我正在使用的API的速率限制器，它允许每秒20个请求。所有请求都是基于承诺的，一旦有回复，承诺将通过API数据得到解决。
I've got a rate limiter for an API I am using which allows 20 requests per second. All requests are promise based and the promise will be resolved with the API data once there is a response.
问题：
我设置了一个promiseArray，它包含所有等待响应的58k promises。所以内存越来越慢，直到我的内存耗尽为止。在我的具体情况下，我不需要将已解析的数据传递给我的 then（），数据占用了我所有的RAM。
I setup a promiseArray which contains 58k promises all waiting for a response. So slowly the memory is increasing until I am running out of memory. In my specific situation I don't need to pass the resolved data to my then() and the data is eating up all my RAM.
代码：
}).then(() => {
// 2. Crawl for all clanprofiles from these leaderboards
const promiseArray = []
for (let i = 0; i < clanTags.length; i++) {
// Resolved data from getClanProfile() is eating up all my RAM
const p = backgroundScheduler.getClanProfile(clanTags[i], true)
promiseArray.push(p)
}
return Promise.all(promiseArray)
}).then(() => {
那么有没有办法等待promiseArray被解析而不需要解析数据？
So is there a way to await until the promiseArray is resolved without needing the resolved data?
推荐答案
你将使用一个如果您没有58k承诺，它们的关联异步操作及其结果数据一次激活，则会减少内存量。
You will use a lesser amount of memory if you don't ever have 58k promises, their associated async operations and their result data active at once.
相反y你想要一次运行X操作，然后当一个完成时，你开始下一个同时飞行中的X不超过X并且一次性使用X承诺。
Instead you want to run X operations at once and then when one finishes, you start the next one with never more than X in flight at the same time and never more than X promises in use at once.
您可以尝试使用适当的X值。值为1是顺序操作，但通常可以通过使用更高的值来改善整体端到端操作时间X.如果所有请求都在同一个主机上，那么X可能不超过5-10（因为给定的主机不能同时做很多事情并要求它做的比它一次做的更多只是放慢速度）。
You can experiment with an appropriate value of X. A value of 1 is sequential operations but you can often improve overall end-to-end operation time by using some higher value of X. If all requests are hitting the same host, then X is probably no more than 5-10 (since a given host can't really do a lots of things at once and asking it to do more than it can do at once just slows it down).
如果每个请求都是针对不同的主机，那么您可以使X更高。实验将为您提供峰值内存使用率和总吞吐量的最佳值，具体取决于您的具体情况。
If every request is to a different host, then you may be able to make X higher. Experimentation would give you an optimal value for both peak memory usage and overall throughput and somewhat depends upon your specific circumstances.
Bluebird的 Promise.map（） 有一个并发选项可以为你做这个，但是也有很多方法可以同时在飞行中编码X.
Bluebird's Promise.map() has a concurrency option that will do this for you, but there are also numerous ways to code for only X in flight at the same time.
以下是一些其他编码示例，用于管理一次飞行中的数量：
Here are some other coding examples of managing how many are in flight at a time:
Fire off 1,000,000 requests 100 at a time
如何使它，以便我可以执行在javascript中一次说10个承诺，以防止api通话的费率限制？
如果你不喜欢需要已解析的数据，您可以通过更换它来更快地进行GC：
If you don't need the resolved data, you can allow it to be GCed sooner by replacing it like this:
const p = backgroundScheduler.getClanProfile(clanTags[i], true).then(data => {
return 0; // make resolved value just be a simple number
// so other data is now eligible for GC
});
promiseArray.push(p)
而且，这是一个简单的实现，它在同一时间迭代一个数组，其中不超过X个请求：
And, here's a simple implementation that iterates an array with no more than X requests in flight at the same time:
// takes an array of items and a function that returns a promise
// runs no more than maxConcurrent requests at once
function mapConcurrent(items, maxConcurrent, fn) {
let index = 0;
let inFlightCntr = 0;
let doneCntr = 0;
let results = new Array(items.length);
let stop = false;
return new Promise(function(resolve, reject) {
function runNext() {
let i = index;
++inFlightCntr;
fn(items[index], index++).then(function(val) {
++doneCntr;
--inFlightCntr;
results[i] = val;
run();
}, function(err) {
// set flag so we don't launch any more requests
stop = true;
reject(err);
});
}
function run() {
// launch as many as we're allowed to
while (!stop && inflightCntr < maxConcurrent && index < items.length) {
runNext();
}
// if all are done, then resolve parent promise with results
if (doneCntr === items.length) {
resolve(results);
}
}
run();
});
}
这篇关于Promise.all消耗我所有的RAM的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
