从S3下载文件> 3Gb失败，并出现“SocketTimeoutException：Read timed out”。 [英] Downloading files >3Gb from S3 fails with "SocketTimeoutException: Read timed out"

问题描述

AWS Java SDK 1.9.3

AWS Java SDK 1.9.3

在Java服务器应用程序中从AWS S3下载许多大文件（~3Gb）时，我会不时收到SocketTimeoutException，如下所示：

When downloading many large files (~3Gb) from AWS S3 in Java server app I get SocketTimeoutException from time to time as following:

Caused by: com.amazonaws.AmazonClientException: Unable to store object contents to disk: Read timed out
 at com.amazonaws.services.s3.internal.ServiceUtils.downloadObjectToFile(ServiceUtils.java:270)
 at com.amazonaws.services.s3.internal.ServiceUtils.retryableDownloadS3ObjectToFile(ServiceUtils.java:344)
 at com.amazonaws.services.s3.transfer.TransferManager$2.call(TransferManager.java:737)
 ... 4 more
Caused by: java.net.SocketTimeoutException: Read timed out
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:152)
 at java.net.SocketInputStream.read(SocketInputStream.java:122)
 at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
 at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
 at sun.security.ssl.InputRecord.read(InputRecord.java:509)
 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
 at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
 at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:198)
 at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
 at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
 at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73)
 at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151)
 at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73)
 at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73)
 at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73)
 at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151)
 at java.security.DigestInputStream.read(DigestInputStream.java:161)
 at com.amazonaws.services.s3.internal.DigestValidationInputStream.read(DigestValidationInputStream.java:59)
 at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73)
 at java.io.FilterInputStream.read(FilterInputStream.java:107)
 at com.amazonaws.services.s3.internal.ServiceUtils.downloadObjectToFile(ServiceUtils.java:265)
 ... 6 more

我使用4个线程从S3下载文件，所有线程都使用相同的TransferManager实例。如前所述，我正在使用AWS SDK 1.9.3并且只是不时获得此类例外。代码如下：

I download files from S3 using 4 threads, all threads use the same TransferManager instance. As mantioned before, I'm using AWS SDK 1.9.3 and get such exceptions only from time to time. Code is the following:

...
Download d = transferManager.download(currentBucket, remoteFileName, new File(tmpName));
d.waitForCompletion();
...

这是SDK的问题吗？可能是v1.9.3？

Is it an issue of SDK? Maybe of v1.9.3?

除了增加 socketTimeout ClientConfiguration之外是否有任何解决方案？

推荐答案

以下是我在AWS SDK 1.9.3中的操作方法：

Here is how I can do in AWS SDK 1.9.3:

ClientConfiguration config = new ClientConfiguration(); 
config.setConnectionTimeout(connectionTimeout);
config.setSocketTimeout(readTimeout); 
AmazonS3 s3 = new AmazonS3Client(credentials, config);

这篇关于从S3下载文件> 3Gb失败，并出现“SocketTimeoutException：Read timed out”。的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！