使用 awscli 恢复中断的 s3 下载 [英] Resuming interrupted s3 download with awscli

问题描述

我正在使用 awscli 下载文件:

I was downloading a file using awscli:

$ aws s3 cp s3://mybucket/myfile myfile

但下载被中断(电脑进入睡眠状态).如何继续下载?S3 支持 Range 标头，但 awscli s3 cp 不允许我指定它.

But the download was interrupted (computer went to sleep). How can I continue the download? S3 supports the Range header, but awscli s3 cp doesn't let me specify it.

该文件不可公开访问，因此我无法使用 curl 手动指定标题.

The file is not publicly accessible so I can't use curl to specify the header manually.

推荐答案

awscli 工具中有一个隐藏"命令，它允许对 S3 进行较低级别的访问:s3api.† 用户较少友好(没有 s3://URL 和进度条)但它确实支持 get-object 上的范围说明符:

There is a "hidden" command in the awscli tool which allows lower level access to S3: s3api.† It is less user friendly (no s3:// URLs and no progress bar) but it does support the range specifier on get-object:

   --range  (string) Downloads the specified range bytes of an object. For
   more   information   about   the   HTTP    range    header,    go    to
   http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.

以下是继续下载的方法:

Here's how to continue the download:

$ size=$(stat -f%z myfile) # assumes OS X. Change for your OS
$ aws s3api get-object \
            --bucket mybucket \
            --key myfile \
            --range "bytes=$size-" \
            /dev/fd/3 3>>myfile

您可以使用 pv 作为基本进度条:

You can use pv for a rudimentary progress bar:

$ aws s3api get-object \
            --bucket mybucket \
            --key myfile \
            --range "bytes=$size-" \
            /dev/fd/3 3>&1 >&2 | pv >> myfile

(这种未命名管道混乱的原因是 s3api 在操作结束时将调试消息写入 stdout，污染了您的文件.此解决方案将 stdout 重新绑定到 stderr 并通过别名释放管道以获取常规文件内容.没有 pv 的版本在技术上可以写入 stderr(/dev/fd/2 和 2>)，但如果发生错误，s3api 会写入stderr，然后它会附加到您的文件中.因此，在那里使用专用管道也更安全.)

(The reason for this unnamed pipe rigmarole is that s3api writes a debug message to stdout at the end of the operation, polluting your file. This solution rebinds stdout to stderr and frees up the pipe for regular file contents through an alias. The version without pv could technically write to stderr (/dev/fd/2 and 2>), but if an error occurs s3api writes to stderr, which would then get appended to your file. Thus, it is safer to use a dedicated pipe there, as well.)

† 在 git 中，s3 是瓷器，而 s3api 是管道.

† In git speak, s3 is porcelain, and s3api is plumbing.

这篇关于使用 awscli 恢复中断的 s3 下载的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！