使用 Amazon S3 根据文件名移动文件 [英] Moving file based on filename with Amazon S3

问题描述

需要将基于名称的 Amazon S3 文件移动到名称相同的相应文件夹.这是针对 AWS 上 python 中的自动化脚本.文件名加1.

Need to move Amazon S3 files based on their names into appropriate folders, that are named the same. This is for an automation script in python on AWS. The file names increment by 1.

例如，一个将被称为 my_file_section_a，然后下一个将是 my_file_section_b1，然后下一个将是 my_file_section_b2 等等.这些文件夹将被称为 my_file_section_a、my_file_section_b1 等等.

For instance, one would be called my_file_section_a, then the next would be my_file_section_b1, then the next would be my_file_section_b2 and so forth. The folders would be called my_file_section_a, my_file_section_b1 and so forth.

代码如下:

from __future__import print_function
import boto3
import time, urllib
import json

print("*"*80)
print("Initializing...")
print("*"*80)

s3 = boto3.client('s3')

def lambda_handler(event, context):

    source_bucket = event['Records'[0]['s3']['bucket']['name']
    object_key = event['Records'][0]['s3']['object']['key']
    target_bucket = 'myfilelambdadestination'
    copy_source = {'Bucket': source_bucket, 'Key': object_key}
    print("Source bucket: ", source_bucket)
    print("Target bucket: ", target_bucket)
    print("Log Stream name: ",context.log_stream_name)
    print("Log Group name: ",context.log_group_name)
    print("Request ID: ",context.aws_request_id)
    print("Mem. limits(MB) ", context.memory_limit_in_mb)
    try:
        print("Using waiter to waiting for object to persist through s3 service")
        waiter = s3.get_waiter('object_exists')
        waiter.wait(Bucket=source_bucket, Key = object_key)
        s3.copy_object(Bucket=target_bucket, Key = object_key, CopySource = copy_source)
        return response['ContentType']
    except Exception as err:
        print("Error -" +str(err))
        return e

如何将基于名称的文件移动到名称相同的文件夹中?

How do I have it that the files based on their names are moved to folders which are also named the same?

推荐答案

这是一个 AWS Lambda 函数，它将一个对象复制到同名目录，然后删除原始对象.

Here is an AWS Lambda function that will copy an object to a directory of the same name, then delete the original object.

它只会从bucket的root复制对象，避免复制的对象再次触发Lambda函数，导致死循环的情况.(Amazon S3 很大，但不是无限大！)如果您的目标是不同的存储桶，则没有必要这样做.

It will only copy objects from the root of the bucket, to avoid a situation where the copied object triggers the Lambda function again, causing an infinite loop. (Amazon S3 is big, but not infinite!) If your target is a different bucket, this will not be necessary.

import boto3
import urllib

def lambda_handler(event, context):
    
    # Get the bucket and object key from the Event
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])
    
    # Only copy objects that were uploaded to the bucket root (to avoid an infinite loop)
    if '/' not in key:
        
        # Copy object
        s3_client = boto3.client('s3')
        s3_client.copy_object(
            Bucket = bucket,
            Key = f"{key}/{key}",
            CopySource= {'Bucket': bucket, 'Key': key}
        )
        
        # Delete source object
        s3_client.delete_object(
            Bucket = bucket,
            Key = key
        )

该函数需要一个具有 GetObject 和 PutObject(可能更多?)权限的 IAM 角色.创建此 AWS Lambda 函数后，在 Amazon S3 存储桶上创建一个事件以在创建对象时触发该函数.

The function requires an IAM Role with permission for GetObject and PutObject (maybe more?) on the bucket. After creating this AWS Lambda function, create an Event on the Amazon S3 bucket to trigger the function when an object is created.

这篇关于使用 Amazon S3 根据文件名移动文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！