将s3上的多个zip文件合并为更少的zip文件 [英] Merge multiple zip files on s3 into fewer zip files

问题描述

我们有一个问题，其中s3目录中的某些文件在〜500MiB范围内，而许多其他文件在KiB和Bytes中.我想将所有的小文件合并到数量较少的大文件中，约500MiB.

We have a problem wherein some of the files in a s3 directory are in ~500MiB range, but many other files are in KiB and Bytes. I want to merge all the small files into fewer bigger files of the order of ~500MiB.

在s3文件夹中重写数据而不是下载，在本地合并并推回s3的最有效方法是什么.我可以使用一些实用程序/aws命令来实现它吗?

What is the most efficient way to rewriting data in an s3 folder instead of having to download, merge on local and push back to s3. Is there some utility/aws command i can use to achieve it?

推荐答案

S3是一种存储服务，没有计算功能.对于您的要求，您需要计算(合并).因此，如果不下载，合并和上传，就无法做自己想要做的事情.

S3 is a storage service and has no compute capability. For what you are asking, you need compute (to merge). So you cannot do what you want without downloading, merging and uploading.

这篇关于将s3上的多个zip文件合并为更少的zip文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！