零件成功完成后，S3分段上传失败 [英] S3 Multi-part Upload fails on completion after parts have successfully been completed

问题描述

我正在尝试将动态生成的数据从内存上传到S3.

I'm trying to upload data generated on the fly from memory to S3.

以下是实际上传零件的代码:

Here's the code for actually uploading the parts:

UploadPartRequest partReq = new UploadPartRequest()
    .withBucketName( this.getBucket() )
    .withKey( this.getKey() )
    .withUploadId( uploadId )
    .withPartNumber( partNumber )
    .withPartSize( partSize )
    .withInputStream( is )
    .withGeneralProgressListener( (ProgressEvent e) -> {
            log.info( "{} {} {} {} {} {} bytes transferred",
            this.getBucket(), this.getKey(), uploadId, partNumber,
            e.getEventType(), e.getBytesTransferred() );
    });
UploadPartResult partRes = this.getClient().uploadPart( partReq );

所有部分均已上传后，日志中会出现以下几行，我认为这意味着所有上传的部分均已成功上传.

After all the parts have been uploaded following lines appear in my logs which I presume to mean that all of the upload parts were all successfully uploaded.

1 TRANSFER_PART_COMPLETED_EVENT 
2 TRANSFER_PART_COMPLETED_EVENT 
3 TRANSFER_PART_COMPLETED_EVENT 
4 TRANSFER_PART_COMPLETED_EVENT 
5 TRANSFER_PART_COMPLETED_EVENT 
6 TRANSFER_PART_COMPLETED_EVENT 
7 TRANSFER_PART_COMPLETED_EVENT 
8 TRANSFER_PART_COMPLETED_EVENT 
9 TRANSFER_PART_COMPLETED_EVENT 
10 TRANSFER_PART_COMPLETED_EVENT
11 TRANSFER_PART_COMPLETED_EVENT
12 TRANSFER_PART_COMPLETED_EVENT
13 TRANSFER_PART_COMPLETED_EVENT
14 TRANSFER_PART_COMPLETED_EVENT
15 TRANSFER_PART_COMPLETED_EVENT
17 TRANSFER_PART_COMPLETED_EVENT
16 TRANSFER_PART_COMPLETED_EVENT

在上传期间，每个上载部分的下一行也都出现了很多情况，这告诉我确实是在为这些上载发送数据.

There were many occurrences of this following line too for each of the upload part while they are still uploading which tells me that it is indeed was sending data for those uploads.

REQUEST_BYTE_TRANSFER_EVENT 8192 bytes transferred

这是我用来最终完成整个上传过程的代码.

And here's the code I'm using to finally complete the entire upload process.

CompleteMultipartUploadRequest cmpur = new CompleteMultipartUploadRequest()
    .withBucketName( this.getBucket() )
    .withKey( this.getKey() )
    .withUploadId( result.getUploadId() )
    .withPartETags( new ArrayList<>( this.getUploadPartResults() ) )
    .withGeneralProgressListener((e) -> {
        log.info( "{} {} {} {} bytes transferred",
        this.getBucket(), this.getKey(),
        e.getEventType(), e.getBytesTransferred() );
    });
this.getClient().completeMultipartUpload( cmpur );

发送完完整的分段上传请求之后，我在日志中得到了以下内容:

Right after sending a request to complete the entire multi-part uploads, I'm getting the following in the logs:

REQUEST_CONTENT_LENGTH_EVENT
CLIENT_REQUEST_STARTED_EVENT
HTTP_REQUEST_STARTED_EVENT
HTTP_REQUEST_COMPLETED_EVENT
CLIENT_REQUEST_FAILED_EVENT

然后线程挂在那里.

我认为CLIENT_REQUEST_FAILED_EVENT表示整个上传失败.我想念什么?

I presume CLIENT_REQUEST_FAILED_EVENT means the entire upload failed. What am I missing?

我尝试增加部分大小，因此整个上载部分只有一个上载的部分，并且成功了.如果零件总数超过1，则它将失败.

I tried increasing the part size so there will be just a single uploaded part for the entire multi-part upload and it was successful. If the total parts is more than 1 then it fails.

只有一个部分时，这里是成功上传的日志.

Here's the logs for successful upload when it's only having a single part.

REQUEST_CONTENT_LENGTH_EVENT 0 bytes transferred
CLIENT_REQUEST_STARTED_EVENT 0 bytes transferred
HTTP_REQUEST_STARTED_EVENT 0 bytes transferred
HTTP_REQUEST_COMPLETED_EVENT 0 bytes transferred
RESPONSE_CONTENT_LENGTH_EVENT 0 bytes transferred
HTTP_RESPONSE_STARTED_EVENT 0 bytes transferred
RESPONSE_BYTE_TRANSFER_EVENT 309 bytes transferred
HTTP_RESPONSE_COMPLETED_EVENT 0 bytes transferred
CLIENT_REQUEST_SUCCESS_EVENT 0 bytes transferred
REQUEST_BYTE_TRANSFER_EVENT 135 bytes transferred

也许，我还能在日志中添加更多内容以查看发生了什么问题?

Perhaps, what else can I put in the logs to get something more to see what went wrong?

推荐答案

我发现这是因为我的零件实际上小于要求的最小大小5MB.

I figured out that it was because of the fact that my parts were actually less than the required minimum size which is 5MB.

https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

我去过该页面几次，但我却忽略了它.真的很容易忽略.日志也无济于事.

I've been to that page several times and yet I overlooked it. It's really easy to overlook. The logs didn't help either.

这篇关于零件成功完成后，S3分段上传失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！