使用 AWS Kinesis Firehose 写入 S3 存储桶中的特定文件夹 [英] Write to a specific folder in S3 bucket using AWS Kinesis Firehose

问题描述

我希望能够根据数据中的内容将数据发送到 kinesis firehose.例如，如果我发送此 JSON 数据:

I would like to be able to send data sent to kinesis firehose based on the content inside the data. For example if I sent this JSON data:

{
   "name": "John",
   "id": 345
}

我想根据 id 过滤数据并将其发送到我的 s3 存储桶的子文件夹，例如:S3://myS3Bucket/345_2018_03_05.使用 Kinesis Firehose 或 AWS Lambda 是否可以做到这一点?

I would like to filter the data based on id and send it to a subfolder of my s3 bucket like: S3://myS3Bucket/345_2018_03_05. Is this at all possible with Kinesis Firehose or AWS Lambda?

我现在能想到的唯一方法是为我的每一个可能的 ID 创建一个 kinesis 流，并将它们指向同一个存储桶，然后将我的事件发送到我的应用程序中的那些流，但我会尽量避免这种情况，因为有很多可能的 ID.

The only way I can think of right now is to resort to creating a kinesis stream for every single one of my possible IDs and point them to the same bucket and then send my events to those streams in my application, but I would like to avoid that since there are many possible IDs.

推荐答案

您可能希望使用 S3 事件通知，每次 Firehose 在您的 S3 存储桶(PUT)中放置新文件时都会触发该通知；S3 事件通知应调用您编写的自定义 lambda 函数，该函数读取 S3 文件的内容并将其拆分并将其写入单独的存储桶，请记住，每个 S3 文件可能会包含许多记录，而不是只有一个.

You probably want to use an S3 event notification that gets fired each time Firehose places a new file in your S3 bucket (a PUT); the S3 event notification should call a custom lambda function that you write that reads the contents of the S3 file and splits it up and writes it out to the separate buckets, keeping in mind that each S3 file is likely going to contain many records, not just one.

https://aws.amazon.com/blogs/aws/s3-事件通知/

这篇关于使用 AWS Kinesis Firehose 写入 S3 存储桶中的特定文件夹的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！