到S3桶之间移动文件的最好方法？ [英] Best way to move files between S3 buckets?

问题描述

我想一些文件从生产水桶每天复制到开发桶。

I'd like to copy some files from a production bucket to a development bucket daily.

例如： 复制productionbucket /料/ feedname /日 到developmentbucket /饲料/ feedname /日

For example: Copy productionbucket/feed/feedname/date to developmentbucket/feed/feedname/date

由于我要的文件在文件夹结构如此之深，太耗费时间去每个文件夹和复制/粘贴。

Because the files I want are so deep in the folder structure, it's too time consuming to go to each folder and copy/paste.

我已经有安装驱动器，每个桶和编写Windows批处理脚本发挥各地，但这是非常缓慢的，这样会将下载的所有文件/文件夹复制到本地服务器，并再次备份。

I've played around with mounting drives to each bucket and writing a windows batch script, but that is very slow and it unnecessarily downloads all the files/folders to the local server and back up again.

推荐答案

由于指出alberge （+1），如今优秀的 AWS命令行界面提供了最灵活的方式与（几乎）所有的东西AWS进行交互 - 它同时涵盖了大部分的服务的API，还设有的更高级别S3命令的与你的使用情况具体处理，请参阅S3的 AWS CLI参考：

Update

As pointed out by alberge (+1), nowadays the excellent AWS Command Line Interface provides the most versatile approach for interacting with (almost) all things AWS - it meanwhile covers most services' APIs and also features higher level S3 commands for dealing with your use case specifically, see the AWS CLI reference for S3:

• 同步 - 可同步目录和S3 prefixes 的你的用例覆盖例2 （更细粒度的用法 - 排除， - 包括和preFIX处理等也可）
  • sync - Syncs directories and S3 prefixes. Your use case is covered by Example 2 (more fine grained usage with --exclude, --include and prefix handling etc. is also available):

    以下sync命令复制S3对象同步在另一个指定的preFIX和铲斗在指定的preFIX和桶对象的对象。 [...]
    

    The following sync command syncs objects under a specified prefix and bucket to objects under another specified prefix and bucket by copying s3 objects. [...]
    

    aws s3 sync s3://mybucket s3://mybucket2
    

         

    

  移动S3桶可以通过 PUT对象的方式来实现的文件 - 复制API （随后删除对象）：

  ---

  Initial Answer

  Moving files between S3 buckets can be achieved by means of the PUT Object - Copy API (followed by DELETE Object):

  此实施PUT操作的创建对象的副本   已存储在Amazon S3中。认沽复制操作是一样的   为执行GET，然后付诸表决。添加请求头，   的X AMZ-复制源，使得PUT操作的源对象复制到   目的地桶。

  This implementation of the PUT operation creates a copy of an object that is already stored in Amazon S3. A PUT copy operation is the same as performing a GET and then a PUT. Adding the request header, x-amz-copy-source, makes the PUT operation copy the source object into the destination bucket.

  有对所有现有的AWS软件开发工具包提供，看到在一个操作复制对象。当然，基于脚本的解决方案会在这里明显的第一选择，所以复制对象使用AWS SDK为Ruby 可能是一个很好的起点;如果你preFER的Python代替，同样可以通过博托也当然可以实现，参见方法 copy_key（）在博托的 S3 A​​PI文档。

  There are respective samples for all existing AWS SDKs available, see Copying Objects in a Single Operation. Naturally, a scripting based solution would be the obvious first choice here, so Copy an Object Using the AWS SDK for Ruby might be a good starting point; if you prefer Python instead, the same can be achieved via boto as well of course, see method copy_key() within boto's S3 API documentation.

  PUT对象只复制文件，所以您需要通过明确地删除文件删除对象后仍然一个成功的复制操作，但这将只是另一个几行一次全面的脚本处理桶和文件名是否到位（有各自的例子还有，如见的Deleting一个对象每次请求）。

  PUT Object only copies files, so you'll need to explicitly delete a file via DELETE Object still after a successful copy operation, but that will be just another few lines once the overall script handling the bucket and file names is in place (there are respective examples as well, see e.g. Deleting One Object Per Request).

  这篇关于到S3桶之间移动文件的最好方法？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！