我们可以使用AWS Glue只是将文件从一个S3文件夹复制到另一个S3文件夹吗? [英] Could we use AWS Glue just copy a file from one S3 folder to another S3 folder?

问题描述

我需要将一个压缩的文件从一个AWS S3文件夹复制到另一个文件夹，并希望进行计划的AWS Glue作业.我找不到如此简单任务的示例.如果您知道答案，请提供帮助.也许答案是使用AWS Lambda或其他AWS工具.

I need to copy a zipped file from one AWS S3 folder to another and would like to make that a scheduled AWS Glue job. I cannot find an example for such a simple task. Please help if you know the answer. May be the answer is in AWS Lambda, or other AWS tools.

非常感谢！

推荐答案

您可以执行此操作，并且可能有使用AWS Glue的原因:如果您已链接Glue作业，并且成功完成以下操作会触发glue_job_#2 glue_job_#1.

You can do this, and there may be a reason to use AWS Glue: if you have chained Glue jobs and glue_job_#2 is triggered on the successful completion of glue_job_#1.

下面的简单Python脚本使用boto3库将文件从一个S3文件夹(source)移到另一个文件夹(target)，并有选择地删除source目录中的原始副本.

The simple Python script below moves a file from one S3 folder (source) to another folder (target) using the boto3 library, and optionally deletes the original copy in source directory.

import boto3

bucketname = "my-unique-bucket-name"
s3 = boto3.resource('s3')
my_bucket = s3.Bucket(bucketname)
source = "path/to/folder1"
target = "path/to/folder2"

for obj in my_bucket.objects.filter(Prefix=source):
    source_filename = (obj.key).split('/')[-1]
    copy_source = {
        'Bucket': bucketname,
        'Key': obj.key
    }
    target_filename = "{}/{}".format(target, source_filename)
    s3.meta.client.copy(copy_source, bucketname, target_filename)
    # Uncomment the line below if you wish the delete the original source file
    # s3.Object(bucketname, obj.key).delete()

参考:Boto3 S3客户端副本上的文档

注意:我将使用f-strings生成target_filename，但是f-strings仅在> = Python3.6中受支持，并且我相信默认的AWS Glue Python解释器仍为2.7.

Note: I would use f-strings for generating the target_filename, but f-strings are only supported in >= Python3.6 and I believe the default AWS Glue Python interpreter is still 2.7.

参考:F弦上的PEP

这篇关于我们可以使用AWS Glue只是将文件从一个S3文件夹复制到另一个S3文件夹吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！