在 Apache Flink 中使用 DynamoDB 流 [英] Consume DynamoDB streams in Apache Flink

问题描述

有没有人尝试在 Apache Flink 中使用 DynamoDB 流?

Has anyone tried to consume DynamoDB streams in Apache Flink ?

Flink 有一个 Kinesis 消费者.但我正在寻找如何直接使用 Dynamo 流.

Flink has a Kinesis consumer. But I am looking for how can i consume the Dynamo stream directly.

DataStream<String> kinesis = env.addSource(new FlinkKinesisConsumer<>(
    "kinesis_stream_name", new SimpleStringSchema(), consumerConfig));

我尝试了很多搜索，但没有找到任何东西.但是在 Flink Jira 板中发现了一个未决请求.所以我猜这个选项还不可用?我有哪些选择?

I tried searching a lot, but did not find anything. However found an open request pending the Flink Jira board. So I guess this option is not available yet ? What alternatives do I have ?

允许 FlinkKinesisConsumer 适应 AWS DynamoDB Streams

推荐答案

UPDATED ANSWER - 2019

FlinkKinesisConsumer 连接器可以现在在实施此 JIRA 票证后处理 DynamoDB 流.

FlinkKinesisConsumer connector can now process a DynamoDB stream after this JIRA ticket is implemented.

更新答案

Apache Flink 似乎没有使用 DynamoDB 流连接器 adapter，所以它可以从 Kinesis 读取数据，但不能从 DynamoDB 读取数据.

It seems that Apache Flink does not use the DynamoDB stream connector adapter, so it can read data from Kinesis, but it can't read data from DynamoDB.

我认为一种选择可能是实现一个应用程序，将数据从 DynamoDB 流写入 Kinesis，然后在 Apache Flink 中从 Kinesis 读取数据并对其进行处理.

I think one option could be implement an app that would write data from DynamoDB streams to Kinesis and then read data from Kinesis in Apache Flink and process it.

另一种选择是为 Apache Flink 实现自定义 DynamoDB 连接器.您可以使用 现有连接器 作为开始点.

Another option would be to implement custom DynamoDB connector for Apache Flink. You can use existing connector as a starting point.

您还可以查看 Apache Spark Kinesis 连接器.但它似乎也有同样的问题.

Also you can take a look at the Apache Spark Kinesis connector. But it seems that it has the same issue as well.

原答案

DynamoDB 有一个 Kinesis 适配器，允许您使用 Kinesis 客户端库.使用 Kinesis 适配器是使用 DynamoDB 更新的推荐方式(根据 AWS).这将为您提供与直接使用 DynamoDB 流(也称为 DynamoDB 低级 API)相同的数据.

DynamoDB has a Kinesis adaptor that allow you to consume a stream of DynamoDB updates using Kinesis Client Library. Using Kinesis adaptor is a recommended way (according to AWS) of consuming updates from DynamoDB. This will give you same data as using DynamoDB stream directly (also called DynamoDB low-level API).

这篇关于在 Apache Flink 中使用 DynamoDB 流的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！