有什么方法可以使用 boto3 将文件直接写入 S3? [英] Any way to write files DIRECTLY to S3 using boto3?

问题描述

我编写了一个 python 脚本来处理非常大的文件(总共只有几个 TB)，我将在 EC2 实例上运行这些文件.之后，我想将处理后的文件存储在 S3 存储桶中.目前，我的脚本首先将数据保存到磁盘，然后将其上传到 S3.不幸的是，考虑到等待实例首先写入磁盘然后上传所花费的额外时间，这将是相当昂贵的.

I wrote a python script to process very large files (few TB in total), which I'll run on an EC2 instance. Afterwards, I want to store the processed files in an S3 bucket. Currently, my script first saves the data to disk and then uploads it to S3. Unfortunately, this will be quite costly given the extra time spent waiting for the instance to first write to disk and then upload.

有什么办法可以使用boto3将文件直接写入S3存储桶吗?

Is there any way to use boto3 to write files directly to an S3 bucket?

为了澄清我的问题，我问我是否在内存中有一个对象，将该对象直接写入 S3，而无需先将该对象保存到磁盘上.

to clarify my question, I'm asking if I have an object in memory, writing that object directly to S3 without first saving the object onto disk.

推荐答案

您可以使用 put_object 为此.只需将您的文件对象作为正文传递即可.

You can use put_object for this. Just pass in your file object as body.

例如:

import boto3

client = boto3.client('s3')
response = client.put_object( 
    Bucket='your-s3-bucket-name',
    Body='bytes or seekable file-like object',
    Key='Object key for which the PUT operation was initiated'
)

这篇关于有什么方法可以使用 boto3 将文件直接写入 S3?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！