如何从 S3 加载泡菜文件以在 AWS Lambda 中使用? [英] How to load a pickle file from S3 to use in AWS Lambda?

问题描述

我目前正在尝试将 S3 中的 pickle 文件加载到 AWS lambda 中并将其存储到列表中(pickle 是一个列表).

I am currently trying to load a pickled file from S3 into AWS lambda and store it to a list (the pickle is a list).

这是我的代码:

import pickle
import boto3

s3 = boto3.resource('s3')
with open('oldscreenurls.pkl', 'rb') as data:
    old_list = s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)

即使文件存在，我也收到以下错误:

I get the following error even though the file exists:

FileNotFoundError: [Errno 2] No such file or directory: 'oldscreenurls.pkl'

有什么想法吗?

推荐答案

如download_fileobj，需要先以二进制write方式打开文件并保存到文件中.下载文件后，您可以打开它进行阅读和解压.

As shown in the documentation for download_fileobj, you need to open the file in binary write mode and save to the file first. Once the file is downloaded, you can open it for reading and unpickle.

import pickle
import boto3

s3 = boto3.resource('s3')
with open('oldscreenurls.pkl', 'wb') as data:
    s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)

with open('oldscreenurls.pkl', 'rb') as data:
    old_list = pickle.load(data)

download_fileobj 获取 S3 中对象的名称加上本地文件的句柄，并将该对象的内容保存到文件中.这个函数还有一个版本叫做 download_file 使用文件名而不是打开的文件句柄并为您打开它.

download_fileobj takes the name of an object in S3 plus a handle to a local file, and saves the contents of that object to the file. There is also a version of this function called download_file that takes a filename instead of an open file handle and handles opening it for you.

在这种情况下，最好使用 S3Client.get_object ，以避免必须写入然后立即读取文件.您还可以写入内存中的 BytesIO 对象，它的作用类似于文件，但实际上并不接触磁盘.看起来像这样:

In this case it would probably be better to use S3Client.get_object though, to avoid having to write and then immediately read a file. You could also write to an in-memory BytesIO object, which acts like a file but doesn't actually touch a disk. That would look something like this:

import pickle
import boto3
from io import BytesIO

s3 = boto3.resource('s3')
with BytesIO() as data:
    s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)
    data.seek(0)    # move back to the beginning after writing
    old_list = pickle.load(data)

这篇关于如何从 S3 加载泡菜文件以在 AWS Lambda 中使用?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！