将文件从私有S3存储桶读取到 pandas 数据框 [英] Reading a file from a private S3 bucket to a pandas dataframe

问题描述

我正在尝试将CS​​V文件从私有S3存储桶读取到熊猫数据框:

I'm trying to read a CSV file from a private S3 bucket to a pandas dataframe:

df = pandas.read_csv('s3://mybucket/file.csv')

我可以从公共存储桶中读取文件，但是从私有存储桶中读取文件会导致HTTP 403:禁止错误.

I can read a file from a public bucket, but reading a file from a private bucket results in HTTP 403: Forbidden error.

我已经使用aws configure配置了AWS凭证.

I have configured the AWS credentials using aws configure.

我可以使用使用aws凭证的boto3从专用存储桶中下载文件.似乎我需要将熊猫配置为使用AWS凭证，但不知道如何.

I can download a file from a private bucket using boto3, which uses aws credentials. It seems that I need to configure pandas to use AWS credentials, but don't know how.

推荐答案

Pandas在read_csv中使用boto(而不是boto3).您也许可以安装boto并使其正常工作.

Pandas uses boto (not boto3) inside read_csv. You might be able to install boto and have it work correctly.

boto和python 3.4.4/python3.5.1存在一些麻烦.如果您使用的是这些平台，并且在修复这些平台之前，可以将boto 3用作

There's some troubles with boto and python 3.4.4 / python3.5.1. If you're on those platforms, and until those are fixed, you can use boto 3 as

import boto3
import pandas as pd

s3 = boto3.client('s3')
obj = s3.get_object(Bucket='bucket', Key='key')
df = pd.read_csv(obj['Body'])

obj具有.read方法(该方法返回字节流)，对于熊猫来说足够了.

That obj had a .read method (which returns a stream of bytes), which is enough for pandas.

这篇关于将文件从私有S3存储桶读取到 pandas 数据框的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！