如何使用boto3快速提取S3中的文件? [英] How to extract files in S3 on the fly with boto3?

问题描述

我正在尝试寻找一种在S3中即时提取.gz文件的方法，无需将其下载到本地，提取然后将其推回S3.

I'm trying to find a way to extract .gz files in S3 on the fly, that is no need to download it to locally, extract and then push it back to S3.

使用boto3 + lambda，我如何实现我的目标?

With boto3 + lambda, how can i achieve my goal?

我在boto3文档中没有看到任何摘录部分.

I didn't see any extract part in boto3 document.

推荐答案

Amazon S3是一项存储服务.没有内置的功能来处理文件的内容.

Amazon S3 is a storage service. There is no in-built capability to manipulate the content of files.

但是，您可以使用AWS Lambda函数从S3检索对象，解压缩对象，然后再次上传内容.但是，请注意，Lambda的临时磁盘空间限制为500MB，因此请避免解压缩太多数据.

However, you could use an AWS Lambda function to retrieve an object from S3, unzip it, then upload content back up again. However, please note that there is limit of 500MB in temporary disk space for Lambda, so avoid unzipping too much data.

您可以将S3存储桶配置为在存储桶中创建新文件时触发Lambda函数.然后，Lambda函数将:

You could configure the S3 bucket to trigger the Lambda function when a new file is created in the bucket. The Lambda function would then:

• 使用boto3(假设您喜欢Python)下载新文件
• 使用zipfile Python库提取文件
• 使用boto3上传生成的文件

• Use boto3 (assuming you like Python) to download the new file
• Use the zipfile Python library to extract files
• Use boto3 to upload the resulting file(s)

示例代码

import boto3

s3 = boto3.client('s3', use_ssl=False)
s3.upload_fileobj(
    Fileobj=gzip.GzipFile(
        None,
        'rb',
        fileobj=BytesIO(
            s3.get_object(Bucket=bucket, Key=gzip_key)['Body'].read())),
    Bucket=bucket,
    Key=uncompressed_key)

这篇关于如何使用boto3快速提取S3中的文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！