是否有可能进行批量上传到Amazon S3？ [英] Is it possible to perform a batch upload to amazon s3?

问题描述

是否Amazon S3的支持批量上传？ 我有一个需要上传的工作，每天晚上〜文件的100K，可以达到1G，但强烈偏向小文件（90％小于100字节和99％，均小于1000字节）。

是否S3 API的支持在单个HTTP调用上传多个对象？

所有的对象都必须在S3可作为单独的对象。我不能承载它们在其他地方（FTP等），或在另一种格式（数据库，EC2本地驱动器等）。这是一个外部的要求，我不能改变。

解决方案

  

是否S3 API的支持在单个HTTP调用上传多个对象？

没有，的S3 PUT操作只支持上传的每个HTTP请求的一个对象。

您可以安装要与远程斗同步 S3工具的你的机器上，并运行以下命令：

  s3cmd同步localdirectory S3：//桶/
 

然后，你可以在脚本中把这个命令，并创建一个计划的作业每天晚上运行此命令。

这应该做你想要什么。

该工具执行基于MD5哈希和文件大小的文件同步，所以碰撞应该是罕见的（如果你真的想你可以只用s3cmd把命令来强制对象的盲覆盖在你的目标桶）。

编辑：另外，还要确保你在网站上阅读我挂了S3工具的文件 - 有需要的您是否要在本地删除的文件被从桶中删除或忽略不同的标志等。

。

Does amazon s3 support batch uploads? I have a job that needs to upload each night ~100K of files that can be up to 1G but is strongly skewed towards small files (90% are less than 100 bytes and 99% are less than 1000 bytes long).

Does the s3 API support uploading multiple objects in a single HTTP call?

All the objects must be available in S3 as individual objects. I cannot host them anywhere else (FTP, etc) or in another format (Database, EC2 local drive, etc). That is an external requirement that I cannot change.

解决方案

Does the s3 API support uploading multiple objects in a single HTTP call?

No, the S3 PUT operation only supports uploading one object per HTTP request.

You could install S3 Tools on your machine that you want to synchronize with the remote bucket, and run the following command:

s3cmd sync localdirectory s3://bucket/

Then you could place this command in a script and create a scheduled job to run this command each night.

This should do what you want.

The tool performs the file synchronization based on MD5 hashes and filesize, so collision should be rare (if you really want you could just use the "s3cmd put" command to force blind overwriting of objects in your target bucket).

EDIT: Also make sure that you read the documentation on the site I linked for S3 Tools - there are different flags needed for whether you want files deleted locally to be deleted from the bucket or ignored etc.

这篇关于是否有可能进行批量上传到Amazon S3？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！