问题上传大文件到Amazon S3时， [英] Problems when uploading large files to Amazon S3
问题描述
我试图使用Amazon的SDK（Java）的样品code称为 S3TransferProgressSample.java上传大文件到Amazon S3 存储（也张贴在这里<一href="http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?HLuploadFileJava.html">http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?HLuploadFileJava.html).但是，当我试图上传11 GB的文件上传卡住的不同点，并显示错误消息无法上传文件到Amazon S3：无法上传部分：无法toexecute HTTP请求：无缓冲实体内附的要求不能重复 （附screesnhot）。它看起来像发生后IOException异常SDK是无法重试的要求（见下文）。有没有人遇到这样的，什么是最好的做法来解决此问题？任何code是AP preciated
I tried to use Amazon SDK (Java) sample code called S3TransferProgressSample.java to upload large files to Amazon s3 storage (also posted here http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?HLuploadFileJava.html). But when I am trying to upload 11 GB files the upload is getting stuck at different points with the error message "Unable to upload file to Amazon S3: Unable to upload part: Unable toexecute HTTP request: Unbuffered entity enclosing request can not be repeated " (attached screesnhot). It looks like after IOException occurs SDK is not able to retry the request (see below). Does anyone encounter this and what is the best-practise to resolve this? Any code is appreciated
信息：收到成功响应：200，AWS请求ID：
2B66E7669E24DA75
2011年1月15日上午06时44分46秒
com.amazonaws.http.HttpClient执行
信息：发送请求：PUT
s3.amazonaws.com /test_file_upload/autogenerated.txt参数：
（uploadId：
m9MqxzD484Ys1nifnX._IzJBGbCFIoT_zBg0xdd6kkZ4TAtmcG0lXQOE.LeiSEuqn6NjcosIQLXJeKzSnKllmw--，型号：1494，）
2011年1月15日上午06时45分10秒
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
信息：I / O异常（产生java.net.SocketException）处理请求时陷入：将连接复位：套接字写入错误
2011年1月15日上午06点45分10秒
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
信息：重试请求
2011年1月15日上午6点45分12秒
com.amazonaws.http.HttpClient执行
警告：无法执行
HTTP请求：无缓冲实体内附请求不能被
反复。照片2011年1月15日上午06点45分12秒
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
信息：I / O异常（产生java.net.SocketException）处理请求时陷入：将连接复位：套接字写入错误
2011年1月15日上午06时45分十二秒
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
信息：重试请求
2011年1月15日上午六时45分13秒
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
信息：I / O异常（产生java.net.SocketException）处理请求时陷入：将连接复位：套接字写入错误
2011年1月15日上午06点45分13秒
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
信息：重试请求
2011年1月15日上午六时45分13秒
com.amazonaws.http.HttpClient执行
警告：无法执行HTTP请求：无缓冲实体内附的要求不能重复
2011年1月15号上午六时45分14秒
com.amazonaws.http.HttpClient执行
警告：无法执行
HTTP请求：无缓冲实体内附请求不能被
反复。照片2011年1月15日上午06时45分14秒com.amazonaws.http.HttpClient
执行
警告：无法执行HTTP请求：无缓冲的实体
封闭请求不能被重复。照片二〇一一年一月十五日上午06时45分14秒
com.amazonaws.http.HttpClient执行
警告：无法执行
HTTP请求：无缓冲实体内附请求不能被
反复。照片2011年1月15日上午6点45分15秒com.amazonaws.http.HttpClient
执行
警告：无法执行HTTP请求：无缓冲的实体
封闭请求不能被重复。照片二○一一年一月十五日上午06点45分16秒
com.amazonaws.http.HttpClient执行
警告：无法执行
HTTP请求：无缓冲实体内附请求不能被
反复。照片2011年1月15日上午06时45分十六秒com.amazonaws.http.HttpClient
执行
警告：无法执行HTTP请求：无缓冲的实体
封闭请求不能被重复。照片二零一一年一月十五日上午06点45分17秒
com.amazonaws.http.HttpClient执行
警告：无法执行
HTTP请求：无缓冲实体内附请求不能被
反复。照片2011年1月15日上午6点45分19秒com.amazonaws.http.HttpClient
执行
警告：无法执行HTTP请求：无缓冲的实体
封闭请求不能被重复。照片二零一一年一月十五日上午06点45分19秒
com.amazonaws.http.HttpClient执行
....
2011年1月15日
上午06时45分21秒com.amazonaws.http.HttpClient用handleResponse
信息：收到成功响应：204，AWS请求ID：E794B8FCA4C3D007
2011年1月15日上午06时45分21秒
com.amazonaws.http.HttpClient执行
...
2011年1月15日6时45分十九秒
AM com.amazonaws.http.HttpClient执行
信息：发送请求：
DELETE s3.amazonaws.com /test_file_upload/autogenerated.txt
参数：
...
2011年1月15日上午06点47分零一秒
com.amazonaws.http.HttpClient handleErrorResponse
信息：收到
错误响应：状态code：404，AWS请求ID：0CE25DFE767CC595，
AWS错误code：NoSuchUpload，AWS错误消息：指定上传
不存在。上载的ID可能是无效的，或上载可以具有
被中止或已完成。
INFO: Received successful response: 200, AWS Request ID: 2B66E7669E24DA75
Jan 15, 2011 6:44:46 AM com.amazonaws.http.HttpClient execute
INFO: Sending Request: PUT s3.amazonaws.com /test_file_upload/autogenerated.txt Parameters: (uploadId: m9MqxzD484Ys1nifnX._IzJBGbCFIoT_zBg0xdd6kkZ4TAtmcG0lXQOE.LeiSEuqn6NjcosIQLXJeKzSnKllmw--, partNumber: 1494, )
Jan 15, 2011 6:45:10 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error
Jan 15, 2011 6:45:10 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
Jan 15, 2011 6:45:12 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:12 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error
Jan 15, 2011 6:45:12 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
Jan 15, 2011 6:45:13 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error
Jan 15, 2011 6:45:13 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
Jan 15, 2011 6:45:13 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:15 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:16 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:16 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:17 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient execute
....
Jan 15, 2011 6:45:21 AM com.amazonaws.http.HttpClient handleResponse
INFO: Received successful response: 204, AWS Request ID: E794B8FCA4C3D007
Jan 15, 2011 6:45:21 AM com.amazonaws.http.HttpClient execute
...
Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient execute
INFO: Sending Request: DELETE s3.amazonaws.com /test_file_upload/autogenerated.txt Parameters:
...
Jan 15, 2011 6:47:01 AM com.amazonaws.http.HttpClient handleErrorResponse
INFO: Received error response: Status Code: 404, AWS Request ID: 0CE25DFE767CC595, AWS Error Code: NoSuchUpload, AWS Error Message: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
感谢您！
-Alexey
推荐答案
尝试使用底层API 。
这会给你远远更多的控制，当事情出错，因为它们很可能做一个11GB的文件。
This will give you far more control when things go wrong, as they are likely to do with an 11GB file.
要求，并从S3做不成不时。与低级别的API，你就可以如果失败重试上载的一部分。
Requests to and from S3 do fail from time to time. With the low level API, you'll be able to retry a part of the upload if it fails.
重构的例子在亚马逊的文档了一下：
Refactoring the example in the Amazon docs a bit:
// Step 2: Upload parts.
long filePosition = 0;
for (int i = 1; filePosition < contentLength; i++) {
// Last part can be less than 5 MB. Adjust part size.
partSize = Math.min(partSize, (contentLength - filePosition));
// Create request to upload a part.
UploadPartRequest uploadRequest = new UploadPartRequest()
.withBucketName(existingBucketName).withKey(keyName)
.withUploadId(initResponse.getUploadId()).withPartNumber(i)
.withFileOffset(filePosition)
.withFile(file)
.withPartSize(partSize);
// repeat the upload until it succeeds.
boolean anotherPass;
do {
anotherPass = false; // assume everythings ok
try {
// Upload part and add response to our list.
partETags.add(s3Client.uploadPart(uploadRequest).getPartETag());
} catch (Exception e) {
anotherPass = true; // repeat
}
}
} while (anotherPass);
filePosition += partSize;
}
// Step 3: complete.
CompleteMultipartUploadRequest compRequest = new
CompleteMultipartUploadRequest(
existingBucketName,
keyName,
initResponse.getUploadId(),
partETags);
s3Client.completeMultipartUpload(compRequest);
注：我不是一个Java开发人员，所以我可以把事情搞乱语法，但希望这可以让你在正确的方向前进。另外，你要在一个重试计数器为prevent添加一个无限循环如果上传失败重复。
Note: I am not a java developer so I could have messed things up syntactically, but hopefully this gets you going in the right direction. Also, you'll want to add in a 'retry counter' to prevent an endless loop if the upload repeatedly fails.
这篇关于问题上传大文件到Amazon S3时，的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
