如何在某些后缀s3中从一个存储桶复制到另一个存储桶 [英] How to copy from one bucket to another bucket in s3 of certain suffix

问题描述

我有3个水桶1.commonfolder 2.jsonfolder 3.csvfolder.

• 公用文件夹将同时包含json和csv文件

• Common folder will be having both json and csv files

需要将所有csv文件复制到csvfolder

need to copy all csv files to csvfolder

需要将所有json文件复制到json文件夹

need to copy all json files to json folder

下面的代码可用于获取commonfolder之后的所有文件

Code is below to get all the files from commonfolder How to copy after that

import boto3

s3 = boto3.client('s3')
def lambda_handler(event, context):
    #List all the bucket names
    response = s3.list_buckets()
    for bucket in response['Buckets']:
        print (bucket)
        print(f'{bucket["Name"]}')
        #Get the files of particular bucket
        if bucket["Name"] == 'tests3json':
         
            resp = s3.list_objects_v2(Bucket='commonfolder')
            for obj in resp['Contents']:
                files = obj['Key']
                print(files)

            if(filename.split('.')[1].lower()=='json'):
                copyjson(bucket,filename)
                #copyjson(jsonfolder,filename)
            elif(filename.split('.')[1].lower()=='csv'):
                copycsv(bucket, filename)
                #copycsv(csvfolder,filename)

• 需要创建一个新函数copyjson，copycsv来完成这项工作

  • need to create a new function copyjson,copycsv to do this job

    需要根据文件扩展名从common-bucket复制到csv-bucket或json-bucket

    Need to copy from common-bucket to either csv-bucket or json-bucket depending on the file extension

    推荐答案

    您可以检查以下代码:

    import boto3
    
    s3 = boto3.resource('s3')
    
    def lambda_handler(event, context):
        
        source_bucket = s3.Bucket('01-commonfolder-231')
        json_bucket = s3.Bucket('02-jsonfolder-3435')
        csv_bucket = s3.Bucket('03-csvfolder-4552')
        
        for object in source_bucket.objects.all():
            
            #print(object)
            
            if object.key.endswith('.json'):
                
                print(f"{object.key} to json bucket")
          
                copy_object = json_bucket.Object(object.key)
                copy_object.copy({'Bucket': object.bucket_name,
                                  'Key': object.key})                             
                
            elif object.key.endswith('.csv'):
                
                print(f"{object.key} to csv bucket")            
    
                copy_object = csv_bucket.Object(object.key)
                copy_object.copy({'Bucket': object.bucket_name,
                                  'Key': object.key})
    

    我使用自己的带有测试文件的样本存储桶对此进行了测试:

    I tested this using my own sample buckets with test files:

    aaa.json to json bucket
    bbbbb.csv to csv bucket
    bbbbb.json to json bucket
    hhhh.csv to csv bucket
    

    这篇关于如何在某些后缀s3中从一个存储桶复制到另一个存储桶的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！