带有AWS Kinesis的Debezium嵌入式引擎-PostgreSQL快照加载和事务元数据流 [英] Debezium Embedded Engine with AWS Kinesis - PostgreSQL snapshot load and Transaction metadata stream

问题描述

我想在AWS Kinesis中使用Debezium嵌入式引擎，以便加载PostgreSQL数据库的初始快照，然后连续执行CDC.

I'd like to use Debezium Embedded Engine with AWS Kinesis in order to load initial snapshot of PostgreSQL database and then continuously perform a CDC.

我知道，使用Kafka Connect，我可以开箱即用的交易元数据"主题，以便检查交易边界.

I know, that with Kafka Connect I'll have Transaction metadata topic out of the box in order to check transaction boundaries.

使用Debezium嵌入式引擎和AWS Kinesis进行同样的操作( https://debezium.io/blog/2018/08/30/streaming-mysql-data-changes-into-kinesis/)在这种情况下，我是否可以使用Kinesis Transaction元数据流?另外，Debezium嵌入式引擎会为现有PostgreSQL数据执行初始快照吗?

How about the same but with Debezium Embedded Engine and AWS Kinesis ( https://debezium.io/blog/2018/08/30/streaming-mysql-data-changes-into-kinesis/ ) Will I have Kinesis Transaction metadata stream in this case? Also, will Debezium Embedded Engine perform initial snapshot of the existing PostgreSQL data?

已更新

我用PostgreSQL实现了测试EmbeddedEngine应用程序:

I implemented test EmbeddedEngine application with PostgreSQL:

    engine = EmbeddedEngine.create()
            .using(config)
            .using(this.getClass().getClassLoader())
            .using(Clock.SYSTEM)
            .notifying(this::sendRecord)
            .build();

现在，在我的'sendRecord(SourceRecord record)'方法中，我可以看到参与事务的每个数据库表的正确主题，例如:

Right now, inside my 'sendRecord(SourceRecord record)' method I can see the correct topics for each database table which participate in transaction, for example:

private void sendRecord(SourceRecord record) {

    String streamName = streamNameMapper(record.topic());
    System.out.println("streamName: " + streamName);

结果为以下输出:

streamName: kinesis.public.user_states
streamName: kinesis.public.tasks

在同一 txId = 1510

，但是我仍然看不到交易元数据流.如何使用Debezium EmbeddedEngine正确获取事务元数据流?

but I still can't see Transaction metadata stream. How to correctly get Transaction metadata stream with Debezium EmbeddedEngine?

推荐答案

如果您不只使用Debezium嵌入式引擎，那么Debezium本身提供了一个选项，它称为Dewbezium Server(内部，我相信它会使用Debezium引擎).它是使用Kafka的一个很好的选择，并且它现在支持CDC的Kinesis，Google PubSub，Apache Pulsar.

If you are not specific about using just Debezium Embedded Engine then there is an option provided by Debezium itself and it is called Dewbezium Server( Internally I believe it makes use of Debezium Engine). It is a good alternative to making use of Kafka and it supports Kinesis, Google PubSub, Apache Pulsar as of now for CDC.

这里是您可以参考的文章 https://xyzcoder.github.io/2021/02/19/cdc-using-debezium-server-mysql-kinesis.html

Here is an article that you can refer to https://xyzcoder.github.io/2021/02/19/cdc-using-debezium-server-mysql-kinesis.html

这篇关于带有AWS Kinesis的Debezium嵌入式引擎-PostgreSQL快照加载和事务元数据流的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！