通过有效负载将AWS Kinesis Firehose数据分区到s3 [英] Partitioning AWS Kinesis Firehose data to s3 by payload

问题描述

我正在使用AWS-Kinesis-Firehose将数据注入S3，然后与Athena一起使用.

I am using AWS-Kinesis-Firehose to injest data to S3, and consume it afterwards with Athena.

我正在尝试分析来自不同游戏的事件，以避免Athena探索大量数据，我想使用每个游戏的标识符对s3数据进行分区，到目前为止，我没有找到解决方案，因为Firehose从不同游戏接收数据.

I am trying to analyze events from different games, to avoid Athena explore much data I would like to partition the s3 data using an identifier for each game, so far I did not find a solution, as Firehose receives data from different games.

有人知道怎么做吗?

谢谢你， 哈维

推荐答案

您可以使用 Amazon Kinesis Analytics 可根据某些逻辑(例如游戏ID)将传入的Firehose流拆分为单独的输出流.

You could possibly use Amazon Kinesis Analytics to split incoming Firehose streams into separate output streams based upon some logic, such as Game ID.

它可以接受 KinesisFirehoseInput 并将数据发送到 KinesisFirehoseOutput .

It can accept a KinesisFirehoseInput and send data to a KinesisFirehoseOutput.

但是，限制文档似乎表明每个应用程序只能有3个输出目标，因此这是不够的.

However, the limits documentation seems to suggest that there can only be 3 output destinations per application, so this would not be sufficient.

这篇关于通过有效负载将AWS Kinesis Firehose数据分区到s3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！