在Node.js中限制对Cassandra DB的并行请求数 [英] limiting number of parallel request to cassandra db in nodejs

问题描述

我目前正在解析文件并获取其数据，以便将它们压入数据库。为此，我创建了一个查询数组，并通过循环执行它们。

I currently parsing a file and getting its data in order tu push them in my db. To do that I made an array of query and I execute them through a loop.

问题是我仅限于2048个并行请求。

The problem is that I'm limited to 2048 parallel requests.

这是我编写的代码：

index.js =>

index.js=>

const ImportClient = require("./scripts/import_client_leasing")
const InsertDb = require("./scripts/insertDb")

const cassandra = require('cassandra-driver');
const databaseConfig = require('./config/database.json');


const authProvider = new cassandra.auth.PlainTextAuthProvider(databaseConfig.cassandra.username, databaseConfig.cassandra.password);

const db = new cassandra.Client({
    contactPoints: databaseConfig.cassandra.contactPoints,
    authProvider: authProvider
});

ImportClient.clientLeasingImport().then(queries => { // this function parse the data and return an array of query
    return InsertDb.Clients(db, queries);    //inserting in the database returns something when all the promises are done
}).then(result => {
    return db.shutdown(function (err, result) {});
}).then(result => {
    console.log(result);
}).catch(error => {
    console.log(error)
});

insertDb.js =>

insertDb.js =>

module.exports = {
    Clients: function (db, queries) {
        DB = db;
        return insertClients(queries);
    }
}

function insertClients(queries) {
    return new Promise((resolve, reject) => {
        let promisesArray = [];

        for (let i = 0; i < queries.length; i++) {
            promisesArray.push(new Promise(function (resolve, reject) {
                DB.execute(queries[i], function (err, result) {
                    if (err) {
                        reject(err)
                    } else {
                        resolve("success");
                    }
                });
            }));
        }
        Promise.all(promisesArray).then((result) => {
            resolve("success");
        }).catch((error) => {
            resolve("error");
        });
    });
}

我尝试了多种方法，例如添加了一个在我的体内设置了timout的await函数每x秒进行一次for循环（但因为我已经在承诺中而没有用），所以我还尝试了 p队列和 p -limit ，但似乎也不起作用。

I tried multiple things, like adding an await function thats set a timout in my for loop every x seconds (but it doesn't work because i'm already in a promise), i also tried with p-queue and p-limit but it doesn't seems to work either.

我有点卡在这里，我想我丢失了一些东西琐碎，但我真的不明白。

I'm kinda stuck here, I'm think I'm missing something trivial but I don't really get what.

谢谢

推荐答案

当并行提交多个请求时（ execute（）函数使用异步执行），最终将以不同级别之一排队：在驱动程序端，在网络堆栈上或在服务器端。过多的排队会影响每个操作完成的总时间。您应该在任何时候限制并发请求的数量，也就是并发级别，以实现高吞吐量和低延迟。

When submitting several requests in parallel (execute() function uses asynchronous execution), you end up queueing at one of the different levels: on the driver side, on the network stack or on the server side. Excessive queueing affects the total time it takes each operation to complete. You should limit the amount of simultaneous requests at any time, also known as concurrency level, to get high throughput and low latency.

考虑在代码中实现它时，您应该考虑使用固定的并发级别作为上限来启动固定数量的异步执行，并且只有在该上限内执行完之后才添加新操作。

When thinking about implementing it in your code, you should consider launching a fixed amount of asynchronous executions, using your concurrency level as a cap and only adding new operations once executions within that cap completed.

以下是一个示例在循环中处理项目时如何限制并发执行的数量： https://github.com/datastax/nodejs-driver/blob/master/examples/concurrent-executions/execute-in-loop.js

Here is an example on how to limit the amount of concurrent executions when processing items in a loop: https://github.com/datastax/nodejs-driver/blob/master/examples/concurrent-executions/execute-in-loop.js

// Launch in parallel n async operations (n being the concurrency level)
for (let i = 0; i < concurrencyLevel; i++) {
  promises[i] = executeOneAtATime();
}

// ...
async function executeOneAtATime() {
  // ...
  // Execute queries asynchronously in sequence
  while (counter++ < totalLength) {;
    await client.execute(query, params, options);
  }
}

这篇关于在Node.js中限制对Cassandra DB的并行请求数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！