使用nodejs对cassandra表上的100,000个用户的记录应用更新查询。忙连接问题 [英] Applying update query on a record of 100,000 users on cassandra table using nodejs. Busy Connection issue

问题描述

我正在使用cassandra 3.x和node 10.13.0。我的工作内存中有100,000个用户的数据（在地图 sortedRowMap中，如给定代码所示）。我将通过迭代每个用户使用的地图（记录为100,000个）来更新所有记录。但这让我感到忙碌。我想知道如何才能摆脱困境。

I'm working with cassandra 3.x and node 10.13.0. I've data of 100,000 users in my working memory (in a map 'sortedRowMap' as in given code). I'm updating all of the record by iterating the map (having record of 100,000) users using for each. But it's throwing me BusyConnectionError. I'm wondering how could I get out from this.

下面是上述说明的实现代码。

Below is the implemented code of the aforementioned description.

var cassClient = new cassandra.Client({contactPoints: ['localhost'],pooling: {
    coreConnectionsPerHost: {
        [distance.local] : 2,
        [distance.remote] : 1
    },
}, keyspace: 'xyz',
    socketOptions: { readTimeout: 65000 }
});

rank = 0;

for (const [msisdn, totalearnings] of sortedRowMap) {

     let updateRankQuery = "UPDATE users SET weeklyrank = " + rank + " WHERE 
     msisdn = " + msisdn;

     cassClient.execute(updateRankQuery, function (error, result) {
            if(!error){
                rank++;
                console.log("updateQuery: " + updateRankQuery)

            }else{
                console.log("ERROR: " + error)
            }
        })

}

并抛出此错误：

ERROR ：NoHostAvailableError：尝试查询的所有主机均失败。尝试了第一个主机127.0.0.1:9042：BusyConnectionError：与主机127.0.0.1:9042的所有连接都处于繁忙状态，每个连接上正在进行2048个请求。请参阅innerErrors。

ERROR: NoHostAvailableError: All host(s) tried for query failed. First host tried, 127.0.0.1:9042: BusyConnectionError: All connections to host 127.0.0.1:9042 are busy, 2048 requests are in-flight on each connection. See innerErrors.

推荐答案

您需要增加每个连接的执行中请求数。协议版本3+支持多达32k的运行中请求。您需要添加相应的选项（ maxRequestsPerConnection ）放入池对象，例如：

You need to increase the number of in-flight requests per connection. Protocol version 3+ support up to 32k in-flight requests. You need to add corresponding option (maxRequestsPerConnection) into pooling object, like, this:

pooling: {
    coreConnectionsPerHost: {
        [distance.local] : 2,
        [distance.remote] : 1
    },
    maxRequestsPerConnection: 32768
}

但是如果所有请求都相同，您仍然可以遇到此异常节点。在这种情况下，您将需要限制提交或重试提交

But you can still hit this exception if all your requests are hitting the same nodes. In this case you'll need to throttle your submissions, or re-try them

另请参见相应的 DataStax开发指南中的部分。

这篇关于使用nodejs对cassandra表上的100,000个用户的记录应用更新查询。忙连接问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！