当最终流大小未知时，将流上传到Google Storage API [英] Streaming upload to Google Storage API when the final stream size is not known

问题描述

因此Google Storage具有出色的可恢复上传的API:

So Google Storage has this great API for resumable uploads: https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload which I'd like to utilize to upload a large object in multiple chunks. However this is done a in stream processing pipeline where the total amount of bytes in the stream is not know in advance.

根据API的文档，您应该使用 Content-Range 标头来告知Google Storage API您已完成文件的上传，例如:

According to the documentation of the API, you're supposed to use Content-Range header to tell the Google Storage API that you're done uploading the file, e.g.:

PUT https://www.googleapis.com/upload/storage/v1/b/myBucket/o?uploadType=resumable&upload_id=xa298sd_sdlkj2 HTTP/1.1
Content-Length: 1024
Content-Range: bytes 1023-2048/2048

[BYTES 1023-2048]

如果我了解这是如何正常工作的，那么 Content-Range 标头的 bytes 1023-2048/2048 值就是Google存储空间确定您的身份的方式上传最后一块数据，完成后即可成功完成可恢复的上传会话.

If I'm understanding how this works correctly, that bytes 1023-2048/2048 value of the Content-Range header is how Google Storage determines that you're uploading the last chunk of data and it can successfully finish the resumable upload session after it's done.

但是，对于我来说，总流大小是未知的，因此我需要继续上传，直到没有更多数据要上传为止.有没有一种使用Google Storage API做到这一点的方法?理想情况下，我希望通过某种方式手动告诉API嘿，我完成了，不要再期望我收到任何其他数据了."

In my case however the total stream size is not known in advance, so I need to keep uploading until there's no more data to upload. Is there a way to do this using the Google Storage API? Ideally I'd like some way of manually telling the API "hey I'm done, don't expect any more data from me".

推荐答案

但是对于我来说，总流大小是未知的，

In my case however the total stream size is not known in advance,

在这种情况下，您需要在PUT请求中发送 Content-Range:字节1023-2048/* .但是请注意，这些请求必须为256KiB的倍数:

In this case you need to send Content-Range: bytes 1023-2048/* in the PUT requests. Note however, that these requests must be in multiples of 256KiB:

https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload#example_uploading_the_file

所以我需要继续上传，直到没有更多数据可以上传为止.是否可以使用Google Storage API做到这一点?

so I need to keep uploading until there's no more data to upload. Is there a way to do this using the Google Storage API?

是的.您以 bytes NNNNN-MMMMM/* 发送块.

理想情况下，我希望通过某种方式手动告诉API嘿，我完成了，不要再期望我收到任何数据了."

Ideally I'd like some way of manually telling the API "hey I'm done, don't expect any more data from me".

您可以执行以下操作:(a)发送不是256KiB倍数的块，或者(b)发送具有字节NNN-MMM/(MMM + 1)的块.也就是说，最后一块包含上载的总大小，并指示它包含最后一个字节.

You do that by either (a) sending a chunk that is not a multiple of 256KiB, or (b) sending a chunk with bytes NNN-MMM/(MMM+1). That is, the last chunk contains the total size for the upload and indicates that it contains the last byte.

这篇关于当最终流大小未知时，将流上传到Google Storage API的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！