从Postgres表到具有更新跟踪的KSQL表 [英] From Postgres table to KSQL table with updates tracking

问题描述

我的任务是将数据从Postgres表传输到KSQL表(以便将来与流联接).假设表有三个记录:

My task is transfer data from Postgres table to KSQL table (for future joins with streams). Let's imagine table has three records:

id | name   | description
-------------------------
1  | name1  | description1
2  | name2  | description2
3  | name3  | description3

通过Kafka JdbcSourceConnector可以轻松实现.但是有一个小问题-表中的数据可能会更改.更改也必须在KTable中.

It is easy to do by means of Kafka JdbcSourceConnector. But there is one little problem - data in table may be changed. Changes must be in KTable too.

根据文档，除批量模式外，没有其他方法可以跟踪更改.但是批量模式绝对使用所有行并将它们插入主题.

According to documentation there is no way to track changes except bulk mode. But bulk mode takes absolutely all rows and inserts them into topic.

我想为连接器设置批量模式.为此主题创建一个KSream.为该流创建一个KTable ...

I thought to set up bulk mode for connector. Create a KSream for that topic. Create a KTable for that stream...

在这里，我不知道该怎么办.如何确保Postgres表中的更改也在KTable中?

And here I do not know what to do. How to make sure changes in Postgres table were in KTable too?

推荐答案

批量模式将起作用，您只需定义流的键，然后新的批量写入将更新相同键的KTable.换句话说，您需要确保数据库中的主键不变

Bulk mode would work, you just define the key of the stream, then new bulk writes will update the KTable of the same key. In other words, you need to ensure the primary keys don't change in your database

或者，Debezium是Kafka Connect的CDC版本.

Alternatively, Debezium is the CDC version of Kafka Connect.

JDBC源未捕获UPDATE查询.

JDBC source doesn't capture UPDATE queries, as you've stated.

Debezium将产生包含修改后的行的先前和新版本的记录

Debezium will produce records that contain previous and new versions of the modified rows

这篇关于从Postgres表到具有更新跟踪的KSQL表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！