按负载将 AWS Kinesis Firehose 数据分区到 s3 [英] Partitioning AWS Kinesis Firehose data to s3 by payload

问题描述

我正在使用 AWS-Kinesis-Firehose 将数据注入 S3，然后使用 Athena 使用它.

I am using AWS-Kinesis-Firehose to injest data to S3, and consume it afterwards with Athena.

我正在尝试分析来自不同游戏的事件，以避免 Athena 探索大量数据我想使用每个游戏的标识符对 s3 数据进行分区，到目前为止我没有找到解决方案，因为 Firehose 接收来自不同游戏的数据.

I am trying to analyze events from different games, to avoid Athena explore much data I would like to partition the s3 data using an identifier for each game, so far I did not find a solution, as Firehose receives data from different games.

有人知道怎么做吗?

谢谢，哈维.

推荐答案

您可以使用 Amazon Kinesis Analytics 根据某些逻辑(例如游戏 ID)将传入的 Firehose 流拆分为单独的输出流.

You could possibly use Amazon Kinesis Analytics to split incoming Firehose streams into separate output streams based upon some logic, such as Game ID.

它可以接受 KinesisFirehoseInput 并将数据发送到一个 KinesisFirehoseOutput.

It can accept a KinesisFirehoseInput and send data to a KinesisFirehoseOutput.

然而，限制文档似乎表明每个应用程序只能有 3 个输出目的地，因此这还不够.

However, the limits documentation seems to suggest that there can only be 3 output destinations per application, so this would not be sufficient.

这篇关于按负载将 AWS Kinesis Firehose 数据分区到 s3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！