com.datastax.driver.core.exceptions.BusyPoolException [英] com.datastax.driver.core.exceptions.BusyPoolException

问题描述

每当我在Cassandra中的表中插入超过1000个数据并按id提取数据时，都会引发以下异常:

Whenever I insert data in table in Cassandra, more than 1000 and fetching the data by id, it throws the following exception:

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.BusyPoolException: [localhost/127.0.0.1] Pool is busy (no available connection and the queue has reached its max size 256)))
    at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:213)
    at com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:49)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:277)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution$1.onFailure(RequestHandler.java:340)
    at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
    at com.google.common.util.concurrent.Futures$ImmediateFuture.addListener(Futures.java:153)
    at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1776)
    at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1713)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.query(RequestHandler.java:299)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:274)
    at com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:117)
    at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:97)
    at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:132)
    at com.outworkers.phantom.builder.query.CassandraOperations$class.scalaQueryStringToPromise(CassandraOperations.scala:67)
    at com.outworkers.phantom.builder.query.InsertQuery.scalaQueryStringToPromise(InsertQuery.scala:31)
    at com.outworkers.phantom.builder.query.CassandraOperations$class.scalaQueryStringExecuteToFuture(CassandraOperations.scala:31)
    at com.outworkers.phantom.builder.query.InsertQuery.scalaQueryStringExecuteToFuture(InsertQuery.scala:31)
    at com.outworkers.phantom.builder.query.ExecutableStatement$class.future(ExecutableQuery.scala:80)
    at com.outworkers.phantom.builder.query.InsertQuery.future(InsertQuery.scala:31)
    at nd.cluster.data.store.Points.upsert(Models.scala:114)

我已经使用PoolingOptions解决了上述问题.

I have solved above issue using PoolingOptions.

val poolingOptions = new PoolingOptions()
    .setConnectionsPerHost(HostDistance.LOCAL, 1, 200)
    .setMaxRequestsPerConnection(HostDistance.LOCAL, 256)
    .setNewConnectionThreshold(HostDistance.LOCAL, 100).setCoreConnectionsPerHost(HostDistance.LOCAL, 200)

  val builder1 =  ContactPoint.local
    .noHeartbeat()
    .withClusterBuilder(_.withoutJMXReporting()
      .withoutMetrics().withPoolingOptions(poolingOptions)).keySpace("nd")

现在，即使使用1l数据，它也可以正常工作.但是我不确定它的效率.谁能帮我吗?

Now it is working even with 1l data. But i am not sure about its efficiency. Could anyone please help me ?

推荐答案

这意味着您正在提交太多请求，而不是在提交更多请求之前等待期货完成.

This means that you are submitting too many requests, and not waiting for the futures to complete before submitting more.

每个连接的默认最大请求数为1024.如果所有连接都超过该数目，则连接池将使一些请求入队，最多256.如果队列已满，则抛出BusyPoolException.当然，您可以增加每个连接的最大请求数和每个主机的最大连接数.但是，真正的解决方案当然是限制您的线程.您可以例如批量提交您的请求，然后等待期货完成，然后再提交更多请求，或者使用信号量来调节未决请求的总数，并确保它们不超过一定数量(理论上，该数目必须保持不变)低于num_hosts * max_connections_per_host * max_requests_per_connection –实际上，我建议不要超过1,000，因为这可能不会为您带来更多的吞吐量).

The default maximum number of requests per connection is 1024. If this number is exceeded for all connections, the connection pool will enqueue some requests, up to 256. If the queue gets full, a BusyPoolException is thrown. Of course you can increase the max number of requests per connection, and the number of max connections per host. But the real solution is of course to throttle your thread. You could e.g. submit your requests by batches of 1,000 and then wait on the futures to complete before submitting more, or use a semaphore to regulate the total number of pending requests and make sure they don't exceed a certain number (in theory, this number must stay below num_hosts * max_connections_per_host * max_requests_per_connection – in practice, I don't suggest going above 1,000 as it probably won't bring you more throughput).

您可能会发现此链接有用.

You may find this links useful.

https://github.com/redisson/redisson/issues/438
https://groups.google.com/a/lists.datastax.com/forum/#！topic/java-driver-user/p3CwOL0kNrs 查看全文