如何使用csv.DictReader读取S3中存储的csv? [英] How do I read a csv stored in S3 with csv.DictReader?

问题描述

我有用于获取AWS S3对象的代码.如何使用Python的csv.DictReader读取此StreamingBody?

I have code that fetches an AWS S3 object. How do I read this StreamingBody with Python's csv.DictReader?

import boto3, csv

session = boto3.session.Session(aws_access_key_id=<>, aws_secret_access_key=<>, region_name=<>)
s3_resource = session.resource('s3')
s3_object = s3_resource.Object(<bucket>, <key>)
streaming_body = s3_object.get()['Body']

#csv.DictReader(???)

推荐答案

代码如下:

import boto3
import csv

# get a handle on s3
s3 = boto3.resource(u's3')

# get a handle on the bucket that holds your file
bucket = s3.Bucket(u'bucket-name')

# get a handle on the object you want (i.e. your file)
obj = bucket.Object(key=u'test.csv')

# get the object
response = obj.get()

# read the contents of the file and split it into a list of lines

# for python 2:
lines = response[u'Body'].read().split()

# for python 3 you need to decode the incoming bytes:
lines = response['Body'].read().decode('utf-8').split()

# now iterate over those lines
for row in csv.DictReader(lines):

    # here you get a sequence of dicts
    # do whatever you want with each line here
    print(row)

您可以在实际代码中对此进行一些压缩，但是我尝试逐步保留它，以显示boto3的对象层次结构.

You can compact this a bit in actual code, but I tried to keep it step-by-step to show the object hierarchy with boto3.

编辑根据您对避免将整个文件读入内存的评论:我没有遇到过这样的要求，所以不能说权威，但是我会尝试包装流以获取文本文件-类似的迭代器.例如，您可以使用编解码器库来替换csv解析上面的部分，例如:

Edit Per your comment about avoiding reading the entire file into memory: I haven't run into that requirement so cant speak authoritatively, but I would try wrapping the stream so I could get a text file-like iterator. For example you could use the codecs library to replace the csv parsing section above with something like:

for row in csv.DictReader(codecs.getreader('utf-8')(response[u'Body'])):
    print(row)

这篇关于如何使用csv.DictReader读取S3中存储的csv?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！