在Apache Flink中使用DynamoDB流 [英] Consume DynamoDB streams in Apache Flink

问题描述

有人尝试过在Apache Flink中使用DynamoDB流吗?

Has anyone tried to consume DynamoDB streams in Apache Flink ?

Flink有一个Kinesis使用者.但是我在寻找如何直接使用Dynamo流的方法.

Flink has a Kinesis consumer. But I am looking for how can i consume the Dynamo stream directly.

DataStream<String> kinesis = env.addSource(new FlinkKinesisConsumer<>(
    "kinesis_stream_name", new SimpleStringSchema(), consumerConfig));

我尝试了很多搜索，但没有找到任何东西.但是找到了一个待处理的请求，等待Flink Jira板.因此，我猜该选项尚不可用?我有什么选择?

I tried searching a lot, but did not find anything. However found an open request pending the Flink Jira board. So I guess this option is not available yet ? What alternatives do I have ?

允许FlinkKinesisConsumer适应AWS DynamoDB流

推荐答案

更新后的答案-2019

FlinkKinesisConsumer 连接器可以现在，在实现此 JIRA票据之后，即可处理DynamoDB流.

FlinkKinesisConsumer connector can now process a DynamoDB stream after this JIRA ticket is implemented.

更新后的答案

Apache Flink似乎没有使用DynamoDB流连接器适配器，因此它可以从Kinesis读取数据，但不能从DynamoDB读取数据.

It seems that Apache Flink does not use the DynamoDB stream connector adapter, so it can read data from Kinesis, but it can't read data from DynamoDB.

我认为一个选择是实现一个应用程序，该应用程序将DynamoDB流中的数据写入Kinesis，然后在Apache Flink中从Kinesis中读取数据并进行处理.

I think one option could be implement an app that would write data from DynamoDB streams to Kinesis and then read data from Kinesis in Apache Flink and process it.

另一个选择是为Apache Flink实现自定义DynamoDB连接器.您可以使用现有连接器作为开始点.

Another option would be to implement custom DynamoDB connector for Apache Flink. You can use existing connector as a starting point.

您还可以查看 Apache Spark Kinesis连接器.但似乎也有相同的问题.

Also you can take a look at the Apache Spark Kinesis connector. But it seems that it has the same issue as well.

原始答案

DynamoDB具有Kinesis适配器，使您可以使用

DynamoDB has a Kinesis adaptor that allow you to consume a stream of DynamoDB updates using Kinesis Client Library. Using Kinesis adaptor is a recommended way (according to AWS) of consuming updates from DynamoDB. This will give you same data as using DynamoDB stream directly (also called DynamoDB low-level API).

这篇关于在Apache Flink中使用DynamoDB流的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！