有什么方法可以使用boto3将文件直接写入S3? [英] Any way to write files DIRECTLY to S3 using boto3?

问题描述

我编写了一个python脚本来处理非常大的文件(总共很少TB)，这些文件将在EC2实例上运行.之后，我想将处理后的文件存储在S3存储桶中.当前，我的脚本首先将数据保存到磁盘，然后将其上传到S3.不幸的是，鉴于等待实例先写入磁盘然后上载所花费的额外时间，这将是非常昂贵的.

I wrote a python script to process very large files (few TB in total), which I'll run on an EC2 instance. Afterwards, I want to store the processed files in an S3 bucket. Currently, my script first saves the data to disk and then uploads it to S3. Unfortunately, this will be quite costly given the extra time spent waiting for the instance to first write to disk and then upload.

是否可以使用boto3将文件直接写入S3存储桶?

Is there any way to use boto3 to write files directly to an S3 bucket?

为澄清我的问题，我要问的是内存中是否有对象，然后将该对象直接写入S3，而无需先将该对象保存到磁盘上.

to clarify my question, I'm asking if I have an object in memory, writing that object directly to S3 without first saving the object onto disk.

推荐答案

您可以使用

You can use put_object for this. Just pass in your file object as body.

例如:

import boto3

client = boto3.client('s3')
response = client.put_object( 
    Bucket='your-s3-bucket-name',
    Body='bytes or seekable file-like object',
    Key='Object key for which the PUT operation was initiated'
)

这篇关于有什么方法可以使用boto3将文件直接写入S3?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！