从 Postgres 表到带有更新跟踪的 KSQL 表 [英] From Postgres table to KSQL table with updates tracking

问题描述

我的任务是将数据从 Postgres 表传输到 KSQL 表(以便将来与流连接).假设表有三条记录:

My task is transfer data from Postgres table to KSQL table (for future joins with streams). Let's imagine table has three records:

id | name   | description
-------------------------
1  | name1  | description1
2  | name2  | description2
3  | name3  | description3

借助 Kafka JdbcSourceConnector 很容易做到.但是有一个小问题——表中的数据可能会发生变化.更改也必须在 KTable 中.

It is easy to do by means of Kafka JdbcSourceConnector. But there is one little problem - data in table may be changed. Changes must be in KTable too.

根据文档，除了批量模式外，无法跟踪更改.但是批量模式绝对获取所有行并将它们插入到主题中.

According to documentation there is no way to track changes except bulk mode. But bulk mode takes absolutely all rows and inserts them into topic.

我想为连接器设置批量模式.为该主题创建一个 KSream.为该流创建一个 KTable...

I thought to set up bulk mode for connector. Create a KSream for that topic. Create a KTable for that stream...

在这里我不知道该怎么办.如何确保 Postgres 表中的更改也在 KTable 中?

And here I do not know what to do. How to make sure changes in Postgres table were in KTable too?

推荐答案

批量模式可以工作，你只需定义流的键，然后新的批量写入将更新相同键的 KTable.换句话说，您需要确保数据库中的主键不会发生变化

Bulk mode would work, you just define the key of the stream, then new bulk writes will update the KTable of the same key. In other words, you need to ensure the primary keys don't change in your database

或者，Debezium 是 Kafka Connect 的 CDC 版本.

Alternatively, Debezium is the CDC version of Kafka Connect.

正如您所说，JDBC 源不捕获 UPDATE 查询.

JDBC source doesn't capture UPDATE queries, as you've stated.

Debezium 将生成包含修改行的先前版本和新版本的记录

Debezium will produce records that contain previous and new versions of the modified rows

这篇关于从 Postgres 表到带有更新跟踪的 KSQL 表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！