" ConnectionPoolTimeoutException"在S3中迭代对象时 [英] "ConnectionPoolTimeoutException" when iterating objects in S3

问题描述

我已经使用aws java API工作了一段时间而没有那么多问题。目前我正在使用1.5.2版本的库。

I've been working for some time with aws java API with not so many problems. Currently I'm using the library 1.5.2 version.

当我使用以下代码迭代文件夹中的对象时：

When I'm iterating the objects inside a folder with the following code:

AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials(MyClass.class.getResourceAsStream("AwsCredentials.properties")));

String s3Key = "folder1/folder2";


String bucketName = Constantes.S3_BUCKET;
String key = s3Key +"/input_chopped/";

ObjectListing  current = s3.listObjects(new ListObjectsRequest()
        .withBucketName(bucketName)
        .withPrefix(key));

boolean siguiente  = true;

while (siguiente) {    

    siguiente &= current.isTruncated();
    contador += current.getObjectSummaries().size();

    for (S3ObjectSummary objectSummary : current.getObjectSummaries()) {        
        S3Object object = s3.getObject(new GetObjectRequest(bucketName, objectSummary.getKey()));
        System.out.println(object.getKey());
    }

    current=s3.listNextBatchOfObjects(current);

}

要点：链接：https://gist.github.com/fgblanch/6038699
我收到以下异常：

Gist: Link: https://gist.github.com/fgblanch/6038699 I'm getting the following exception:

INFO  (AmazonHttpClient.java:358) - Unable to execute HTTP request: Timeout waiting for connection from pool
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
    at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:232)
    at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:199)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:456)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:315)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:199)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2994)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:918)
    at com.madiva.segmentacion.tests.ListaS3.main(ListaS3.java:177)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caught an AmazonClientException, which means the client encountered a serious internal problem while trying to communicate with S3, such as not being able to access the network.
Error Message: Unable to execute HTTP request: Timeout waiting for connection from pool

任何想法如何避免这个错误。它只发生在具有多个对象的文件夹中，在这种情况下，里面有463个文件。谢谢

Any idea how to avoid this error. It only happens in folders with a number of object , in this case there were 463 files inside. Thanks

推荐答案

我发现S3Object为每个对象打开一个连接。即使对象被垃圾收集，也不会解放，因此需要执行object.close（），以释放与池的连接。

I've found that S3Object opens a connection for each object. That are not liberated even if the object is garbage collected so it is needed to execute object.close(), in order to liberate the connection to the pool.

所以更正的代码将是：

 for (S3ObjectSummary objectSummary : current.getObjectSummaries()) {        
        S3Object object = s3.getObject(new GetObjectRequest(bucketName, objectSummary.getKey()));             
        System.out.println(object.getKey());
        object.close();
    }

这篇关于" ConnectionPoolTimeoutException"在S3中迭代对象时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！