在 S3 存储桶之间移动文件的最佳方法? [英] Best way to move files between S3 buckets?

问题描述

我想每天将一些文件从生产存储桶复制到开发存储桶.

例如:复制生产桶/饲料/饲料名称/日期到 developmentbucket/feed/feedname/date

因为我想要的文件在文件夹结构中太深了，去每个文件夹复制/粘贴太费时间了.

我已经尝试将驱动器安装到每个存储桶并编写 Windows 批处理脚本，但这非常慢，而且它不必要地将所有文件/文件夹下载到本地服务器并再次备份.

解决方案

更新

正如 alberge 指出的 (+1)，如今优秀的 AWS 命令​​行界面提供了与(几乎)所有 AWS 交互的最通用方法——同时它涵盖了大多数服务的 API 和功能用于专门处理您的用例的更高级别的 S3 命令，请参阅 S3 的 AWS CLI 参考:

• sync - 同步目录和S3 前缀.您的用例包含在 示例 2(也可以使用 --exclude、--include 和前缀处理等更细粒度的用法):<块引用>

  以下同步命令通过复制 s3 对象将指定前缀和存储桶下的对象同步到另一个指定前缀和存储桶下的对象.[...]

  aws s3 同步 s3://from_my_bucket s3://to_my_other_bucket

为了完整起见，我会提到较低级别的 S3 命令仍然可以通过 s3api 子命令，这将允许在最终采用其更高级别的功能之前，将任何基于 SDK 的解决方案直接转换为 AWS CLI.

<小时>

初始答案

在 S3 存储桶之间移动文件可以通过 PUT Object - Copy 实现API(后跟删除对象):

<块引用>

这个 PUT 操作的实现创建了一个对象的副本已存储在 Amazon S3 中.PUT 复制操作是一样的执行 GET 然后执行 PUT.添加请求头，x-amz-copy-source，使 PUT 操作将源对象复制到目标存储桶.来源

所有现有 AWS 开发工具包都有各自的示例，请参阅复制对象单一操作.自然，基于脚本的解决方案显然是这里的首选，因此 复制对象使用适用于 Ruby 的 AWS SDK 可能是一个很好的起点；如果你更喜欢 Python，当然也可以通过 boto 来实现，参见方法 boto 的 S3 API 文档中的 copy_key().

PUT Object 仅复制文件，因此在成功复制操作后仍需要通过 DELETE Object 显式删除文件，但这只是另外几个一旦处理存储桶和文件名的整体脚本就位(也有相应的示例，请参见例如 每个请求删除一个对象).

I'd like to copy some files from a production bucket to a development bucket daily.

For example: Copy productionbucket/feed/feedname/date to developmentbucket/feed/feedname/date

Because the files I want are so deep in the folder structure, it's too time consuming to go to each folder and copy/paste.

I've played around with mounting drives to each bucket and writing a windows batch script, but that is very slow and it unnecessarily downloads all the files/folders to the local server and back up again.

解决方案

Update

As pointed out by alberge (+1), nowadays the excellent AWS Command Line Interface provides the most versatile approach for interacting with (almost) all things AWS - it meanwhile covers most services' APIs and also features higher level S3 commands for dealing with your use case specifically, see the AWS CLI reference for S3:

• sync - Syncs directories and S3 prefixes. Your use case is covered by Example 2 (more fine grained usage with --exclude, --include and prefix handling etc. is also available):

  The following sync command syncs objects under a specified prefix and bucket to objects under another specified prefix and bucket by copying s3 objects. [...]

  aws s3 sync s3://from_my_bucket s3://to_my_other_bucket
  

For completeness, I'll mention that the lower level S3 commands are also still available via the s3api sub command, which would allow to directly translate any SDK based solution to the AWS CLI before adopting its higher level functionality eventually.

---

Initial Answer

Moving files between S3 buckets can be achieved by means of the PUT Object - Copy API (followed by DELETE Object):

This implementation of the PUT operation creates a copy of an object that is already stored in Amazon S3. A PUT copy operation is the same as performing a GET and then a PUT. Adding the request header, x-amz-copy-source, makes the PUT operation copy the source object into the destination bucket. Source

There are respective samples for all existing AWS SDKs available, see Copying Objects in a Single Operation. Naturally, a scripting based solution would be the obvious first choice here, so Copy an Object Using the AWS SDK for Ruby might be a good starting point; if you prefer Python instead, the same can be achieved via boto as well of course, see method copy_key() within boto's S3 API documentation.

PUT Object only copies files, so you'll need to explicitly delete a file via DELETE Object still after a successful copy operation, but that will be just another few lines once the overall script handling the bucket and file names is in place (there are respective examples as well, see e.g. Deleting One Object Per Request).

这篇关于在 S3 存储桶之间移动文件的最佳方法?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！