Promise.all 消耗了我所有的 RAM [英] Promise.all consumes all my RAM
问题描述
我有一个用于我正在使用的 API 的速率限制器，它允许每秒 20 个请求.所有请求都基于承诺，一旦有响应，承诺将通过 API 数据解决.
问题:
我设置了一个 promiseArray，其中包含 58k 个承诺，所有承诺都在等待响应.如此缓慢地内存增加，直到我的内存不足.在我的特定情况下，我不需要将解析的数据传递给我的 then() 并且数据占用了我所有的 RAM.
代码:
}).then(() => {//2. 从这些排行榜中抓取所有部落档案const promiseArray = []for (let i = 0; i { 那么有没有一种方法可以在不需要解析数据的情况下等待 promiseArray 解析完成?
如果您没有 58k 承诺、它们相关的异步操作和它们的结果数据同时处于活动状态，那么您将使用较少的内存量.
相反，您希望一次运行 X 个操作，然后当一个操作完成时，您开始下一个操作，同时在运行中永远不会超过 X，并且一次使用的承诺永远不会超过 X.
您可以尝试使用适当的 X 值.值为 1 是顺序操作，但您通常可以通过使用更高的 X 值来改善整体端到端操作时间.如果所有请求都访问同一主机，那么 X 可能不超过 5-10(因为一个给定的主机不能一次真正做很多事情，并且要求它做的比它一次做的更多只会减慢它的速度).
如果每个请求都发送到不同的主机，那么您可以将 X 设置得更高.实验将为您提供峰值内存使用量和总吞吐量的最佳值，这在一定程度上取决于您的具体情况.
Bluebird 的 Promise.map()有一个并发选项可以为您执行此操作，但也有多种方法可以同时为 X 进行编码.
以下是管理一次飞行数量的其他一些编码示例:
如何做到这一点，以便我可以在 javascript 中一次执行 10 个承诺以防止 api 调用的速率限制?
如果你不需要解析的数据，你可以通过像这样替换它来让它更快地被GC:
const p = backgroundScheduler.getClanProfile(clanTags[i], true).then(data => {返回0；//使解析值只是一个简单的数字//所以其他数据现在有资格进行 GC});promiseArray.push(p)而且，这是一个简单的实现，它迭代一个数组，同时不超过 X 个请求:
//接受一个项目数组和一个返回承诺的函数//一次运行不超过 maxConcurrent 请求函数 mapConcurrent(items, maxConcurrent, fn) {让索引 = 0;让 inFlightCntr = 0;让 doneCntr = 0;让结果 = 新数组(items.length);让停止=假；返回新的承诺(功能(解决，拒绝){函数运行下一个(){让我 = 指数;++inFlightCntr;fn(items[index], index++).then(function(val) {++doneCntr;--inFlightCntr;结果[i] = val;跑();}, 函数(错误){//设置标志以便我们不再发起任何请求停止 = 真;拒绝(错误)；});}函数运行(){//尽可能多地启动while (!stop && inFlightCntr < maxConcurrent && index < items.length) {运行下一个()；}//如果都完成了，那么用结果解析父承诺if (doneCntr === items.length) {解决(结果)；}}跑();});}I've got a rate limiter for an API I am using which allows 20 requests per second. All requests are promise based and the promise will be resolved with the API data once there is a response.
The problem:
I setup a promiseArray which contains 58k promises all waiting for a response. So slowly the memory is increasing until I am running out of memory. In my specific situation I don't need to pass the resolved data to my then() and the data is eating up all my RAM.
The code:
}).then(() => {
// 2. Crawl for all clanprofiles from these leaderboards
const promiseArray = []
for (let i = 0; i < clanTags.length; i++) {
// Resolved data from getClanProfile() is eating up all my RAM
const p = backgroundScheduler.getClanProfile(clanTags[i], true)
promiseArray.push(p)
}
return Promise.all(promiseArray)
}).then(() => {
So is there a way to await until the promiseArray is resolved without needing the resolved data?
You will use a lesser amount of memory if you don't ever have 58k promises, their associated async operations and their result data active at once.
Instead you want to run X operations at once and then when one finishes, you start the next one with never more than X in flight at the same time and never more than X promises in use at once.
You can experiment with an appropriate value of X. A value of 1 is sequential operations but you can often improve overall end-to-end operation time by using some higher value of X. If all requests are hitting the same host, then X is probably no more than 5-10 (since a given host can't really do a lots of things at once and asking it to do more than it can do at once just slows it down).
If every request is to a different host, then you may be able to make X higher. Experimentation would give you an optimal value for both peak memory usage and overall throughput and somewhat depends upon your specific circumstances.
Bluebird's Promise.map() has a concurrency option that will do this for you, but there are also numerous ways to code for only X in flight at the same time.
Here are some other coding examples of managing how many are in flight at a time:
Make several requests to an API that can only handle 20 request a minute
How to execute promises in series?
unable to complete promises due to out of memory
Fire off 1,000,000 requests 100 at a time
If you don't need the resolved data, you can allow it to be GCed sooner by replacing it like this:
const p = backgroundScheduler.getClanProfile(clanTags[i], true).then(data => {
return 0; // make resolved value just be a simple number
// so other data is now eligible for GC
});
promiseArray.push(p)
And, here's a simple implementation that iterates an array with no more than X requests in flight at the same time:
// takes an array of items and a function that returns a promise
// runs no more than maxConcurrent requests at once
function mapConcurrent(items, maxConcurrent, fn) {
let index = 0;
let inFlightCntr = 0;
let doneCntr = 0;
let results = new Array(items.length);
let stop = false;
return new Promise(function(resolve, reject) {
function runNext() {
let i = index;
++inFlightCntr;
fn(items[index], index++).then(function(val) {
++doneCntr;
--inFlightCntr;
results[i] = val;
run();
}, function(err) {
// set flag so we don't launch any more requests
stop = true;
reject(err);
});
}
function run() {
// launch as many as we're allowed to
while (!stop && inFlightCntr < maxConcurrent && index < items.length) {
runNext();
}
// if all are done, then resolve parent promise with results
if (doneCntr === items.length) {
resolve(results);
}
}
run();
});
}
这篇关于Promise.all 消耗了我所有的 RAM的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
