使用Lambda从S3读取数据 [英] Reading data from S3 using Lambda

问题描述

我在AWS的S3存储桶中存储了一系列json文件.

I have a range of json files stored in an S3 bucket on AWS.

我希望使用AWS lambda python服务来解析此json并将解析的结果发送到AWS RDS MySQL数据库.

I wish to use AWS lambda python service to parse this json and send the parsed results to an AWS RDS MySQL database.

我有一个稳定的python脚本，用于解析和写入数据库.我需要使用lambda脚本来遍历json文件(添加它们时).

I have a stable python script for doing the parsing and writing to the database. I need to lambda script to iterate through the json files (when they are added).

每个json文件都包含一个列表，简单地由results = [content]

Each json file contains a list, simple consisting of results = [content]

我想要的是伪代码:

1. 连接到S3存储桶(jsondata)
2. 读取JSON文件(results)的内容
3. 对此数据执行脚本(results)

1. Connect to the S3 bucket (jsondata)
2. Read the contents of the JSON file (results)
3. Execute my script for this data (results)

我可以列出我拥有的存储桶:

I can list the buckets I have by:

import boto3

s3 = boto3.resource('s3')

for bucket in s3.buckets.all():
    print(bucket.name)

给予:

jsondata

但是我无法访问该存储桶以读取其结果.

But I cannot access this bucket to read its results.

似乎没有read或load函数.

我希望有

for bucket in s3.buckets.all():
   print(bucket.contents)

编辑

我误会了一些东西. Lambda必须自己下载文件，而不是在S3中读取文件.

I am misunderstanding something. Rather than reading the file in S3, lambda must download it itself.

来自在此似乎您必须给lambda一个下载路径，它可以从该路径访问文件本身

From here it seems that you must give lambda a download path, from which it can access the files itself

import libraries

s3_client = boto3.client('s3')

def function to be executed:
   blah blah

def handler(event, context):
    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key'] 
        download_path = '/tmp/{}{}'.format(uuid.uuid4(), key)
        s3_client.download_file(bucket, key, download_path)

推荐答案

您可以使用

You can use bucket.objects.all() to get a list of the all objects in the bucket (you also have alternative methods like filter, page_sizeand limit depending on your need)

这些方法返回带有

These methods return an iterator with S3.ObjectSummary objects in it, from there you can use the method object.get to retrieve the file.

这篇关于使用Lambda从S3读取数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！