将 Dataframe 保存到 csv 直接到 s3 Python [英] Save Dataframe to csv directly to s3 Python

问题描述

我有一个想要上传到新 CSV 文件的 Pandas DataFrame.问题是我不想在将文件传输到 s3 之前将其保存在本地.有没有像 to_csv 这样的方法可以直接将数据帧写入 s3?我正在使用 boto3.
这是我目前所拥有的:

I have a pandas DataFrame that I want to upload to a new CSV file. The problem is that I don't want to save the file locally before transferring it to s3. Is there any method like to_csv for writing the dataframe to s3 directly? I am using boto3.
Here is what I have so far:

import boto3
s3 = boto3.client('s3', aws_access_key_id='key', aws_secret_access_key='secret_key')
read_file = s3.get_object(Bucket, Key)
df = pd.read_csv(read_file['Body'])

# Make alterations to DataFrame

# Then export DataFrame to CSV through direct transfer to s3

推荐答案

您可以使用:

from io import StringIO # python3; python2: BytesIO 
import boto3

bucket = 'my_bucket_name' # already created on S3
csv_buffer = StringIO()
df.to_csv(csv_buffer)
s3_resource = boto3.resource('s3')
s3_resource.Object(bucket, 'df.csv').put(Body=csv_buffer.getvalue())

这篇关于将 Dataframe 保存到 csv 直接到 s3 Python的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！