是否可以批量上传到亚马逊 s3? [英] Is it possible to perform a batch upload to amazon s3?

问题描述

amazon s3 支持批量上传吗?我有一份工作需要每晚上传约 100K 的文件，这些文件可以高达 1G，但强烈倾向于小文件(90% 小于 100 字节，99% 小于 1000 字节长).

Does amazon s3 support batch uploads? I have a job that needs to upload each night ~100K of files that can be up to 1G but is strongly skewed towards small files (90% are less than 100 bytes and 99% are less than 1000 bytes long).

s3 API 是否支持在单个 HTTP 调用中上传多个对象?

Does the s3 API support uploading multiple objects in a single HTTP call?

所有对象都必须作为单独的对象在 S3 中可用.我无法在其他任何地方(FTP 等)或其他格式(数据库、EC2 本地驱动器等)托管它们.这是我无法更改的外部要求.

All the objects must be available in S3 as individual objects. I cannot host them anywhere else (FTP, etc) or in another format (Database, EC2 local drive, etc). That is an external requirement that I cannot change.

推荐答案

s3 API 是否支持在单个 HTTP 调用中上传多个对象?

Does the s3 API support uploading multiple objects in a single HTTP call?

否，S3 PUT 操作仅支持上传一个每个 HTTP 请求的对象.

No, the S3 PUT operation only supports uploading one object per HTTP request.

您可以在要与远程存储桶同步的机器上安装 S3 Tools，然后运行以下命令命令:

You could install S3 Tools on your machine that you want to synchronize with the remote bucket, and run the following command:

s3cmd sync localdirectory s3://bucket/

然后你可以把这个命令放在一个脚本中，并创建一个计划作业来每晚运行这个命令.

Then you could place this command in a script and create a scheduled job to run this command each night.

这应该可以满足您的需求.

This should do what you want.

该工具根据 MD5 哈希值和文件大小执行文件同步，因此冲突应该很少见(如果你真的想要，你可以只使用s3cmd put"命令来强制盲覆盖目标存储桶中的对象).

The tool performs the file synchronization based on MD5 hashes and filesize, so collision should be rare (if you really want you could just use the "s3cmd put" command to force blind overwriting of objects in your target bucket).

还要确保您阅读了我为 S3 工具链接的站点上的文档 - 您是否希望从存储桶中删除本地删除的文件或忽略等需要不同的标志.

Also make sure that you read the documentation on the site I linked for S3 Tools - there are different flags needed for whether you want files deleted locally to be deleted from the bucket or ignored etc.

这篇关于是否可以批量上传到亚马逊 s3?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！