Python/Boto 3:如何从AWS S3检索/下载文件? [英] Python/ Boto 3: How to retrieve/download files from AWS S3?

问题描述

在Python/Boto 3中，发现将文件从S3单独下载到本地可以执行以下操作:

In Python/Boto 3, Found out that to download a file individually from S3 to local can do the following:

        bucket = self._aws_connection.get_bucket(aws_bucketname)
        for s3_file in bucket.list():
            if filename == s3_file.name:
                self._downloadFile(s3_file, local_download_directory)
                break;

并下载一个选定目录下的所有文件:

And to download all files under one chosen directory:

    else:
        bucket = self._aws_connection.get_bucket(aws_bucketname)
        for s3_file in bucket.list():
            self._downloadFile(s3_file, local_download_directory)

和辅助功能_downloadFile():

  def _downloadFile(self, s3_file, local_download_destination):
        full_local_path = os.path.expanduser(os.path.join(local_download_destination, s3_file.name))
        try:
            print "Downloaded: %s" % (full_local_path)
            s3_file.get_contents_to_filename(full_local_path)

但是两者似乎都没有用.使用Boto 3和Python，希望能够将所有文件(最好是zip)下载到S3上已定义目录下的本地文件中.

But both don’t seem to be working. Using Boto 3 and Python, would like to be able to download all files, as a zip preferably, under a defined directory on S3 to my local.

我可能做错了什么?正确的参数实现是什么?

What could I be doing wrong, and what’s the correct implementation of the parameters?

提前谢谢您，一定会接受/支持答案

Thank you in advance, and will be sure to accept/upvote answer

更新代码:Getting an error: AttributeError: 'S3' object has no attribute

import sys
import json
import os
import subprocess

import boto3
from boto.s3.connection import S3Connection

s3 = boto3.resource('s3')
s3client = boto3.client('s3')

#This works
for bucket in s3.buckets.all():
    print(bucket.name)

def main():
    #Getting an error: "AttributeError: 'S3' object has no attribute 'download’"
    s3client.download('testbucket', 'fileone.json', 'newfile')

if __name__ == "__main__": main()

推荐答案

要将文件从S3下载到本地FS，请使用

To download files from S3 to Local FS, use the download_file() method

s3client = boto3.client('s3')
s3client.download_file(Bucket, Key, Filename)

如果S3对象为s3://mybucket/foo/bar/file.txt，则参数为

If the S3 object is s3://mybucket/foo/bar/file.txt, then the arguments would be

Bucket   --> mybucket
Key      --> foo/bar/file.txt
Filename --> /local/path/file.txt

没有任何方法可以下载整个存储桶.另一种方法是列出存储桶中的所有对象，然后将其单独下载为文件.

There aren't any methods to download the entire bucket. An alternative way would be to list all the objects in the bucket and download them individually as files.

for obj in s3client.list_objects(Bucket='mybucket')['Contents']:
    try:  
        filename = obj['Key'].rsplit('/', 1)[1]
    except IndexError:
        filename = obj['Key']

    localfilename = os.path.join('/home/username/Downloads/', filename)  # The local directory must exist.
    s3client.download_file('mybucket', obj['Key'], localfilename)

注意: list_objects() 被截断为1000个对象.在响应中使用标记来检索存储桶中剩余的对象.

Note: The response of list_objects() is truncated to 1000 objects. Use the markers in the response to retrieve the remainder of objects in the bucket.

这篇关于Python/Boto 3:如何从AWS S3检索/下载文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！