将文件列表从S3存储桶复制到S3存储桶 [英] Copy list of files from S3 bucket to S3 bucket

问题描述

是否可以将文件列表从一个S3存储桶复制到另一个存储桶?两个S3存储桶都在同一个AWS账户中.我可以使用aws cli命令一次复制一个文件:

Is there a way I could copy a list of files from one S3 bucket to another? Both S3 buckets are in the same AWS account. I am able to copy a single file at a time using the aws cli command:

     aws s3 cp s3://source-bucket/file.txt s3://target-bucket/file.txt

但是我要复制1000多个文件.我不想复制源存储桶中的所有文件，因此无法使用sync命令.有没有一种方法可以使用需要复制的文件名列表来调用文件，以自动执行此过程?

However I have 1000+ files to copy. I do not want to copy all files in the source bucket so I am not able to utilize the sync command. Is there a way to call a file with the list of file names that needs to be copied to automate this process?

推荐答案

从Python方面解决此问题，您可以运行一个为您执行此操作的Python脚本.由于您有很多文件，因此可能需要一段时间，但应该可以完成工作.将以下代码保存在扩展名为 .py 的文件中并运行它.如果您还没有终端，可能需要事先在终端中运行 pip install boto3 .

Approaching this problem from the Python aspect, you can run a Python script that does it for you. Since you have a lot of files, it might take a while but should get the job done. Save the following code in a file with .py extension and run it. You might need to run pip install boto3 beforehand in your terminal in case you don't already have it.

import boto3
s3 = boto3.resource('s3')
mybucket = s3.Bucket('oldBucket')
list_of_files = ['file1.txt', 'file2.txt']
for obj in mybucket.objects.all():
    if obj.key in list_of_files:
        s3.Object('newBucket', obj.key).put(Body=obj.get()["Body"].read())

这篇关于将文件列表从S3存储桶复制到S3存储桶的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！