Debezium Postgres Kafka Connector心跳未提交LSN [英] Debezium Postgres Kafka Connector heartbeat is not committing LSN
问题描述
我在AWS RDS上有一个Postgres Db,并且有一个kafka connect连接器(Debezium Postgres)在桌上监听.连接器的配置:
I have a Postgres Db on AWS RDS and a kafka connect connector (Debezium Postgres) listening on a table. The configuration of the connector:
{
"name": "my-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.dbname": "my_db",
"database.user": "my_user",
"max.queue.size": "32000",
"slot.name": "my_slot",
"tasks.max": "1",
"publication.name": "my_publication",
"database.server.name": "postgres",
"heartbeat.interval.ms": "1000",
"database.port": "my_port",
"include.schema.changes": "false",
"plugin.name": "pgoutput",
"table.whitelist": "public.my_table",
"tombstones.on.delete": "false",
"database.hostname": "my_host",
"database.password": "my_password",
"name": "my-connector",
"max.batch.size": "10000",
"database.whitelist": "my_db",
"snapshot.mode": "never"
},
"tasks": [
{
"connector": "my-connector",
"task": 0
}
],
"type": "source"
}
该表的更新频率不及其他表,因此最初会导致复制滞后,如下所示:
The table is not updated as frequently as other tables, which initially led to replication lag like this:
SELECT slot_name,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) as replicationSlotLag,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) as confirmedLag,
active
FROM pg_replication_slots;
slot_name | replicationslotlag | confirmedlag | active
-------------------------------+--------------------+--------------+--------
my_slot | 1664 MB | 1664 MB | t
它会变得如此之大,以至于威胁到会耗尽所有磁盘空间.
It would get so large that it would threaten using up all disk space.
我添加了一个心跳信号,并且如果我登录到kafka代理并按以下方式设置控制台使用者: ./kafka-console-consumer.sh --bootstrap-server my.broker.address:9092--topic __debezium-heartbeat.postgres-从头开始--consumer.config =/etc/kafka/consumer.properties
它将转储所有心跳消息,然后每1000毫秒显示一个新消息.
I added a heartbeat, and if I log onto a kafka broker and set up a console consumer like this: ./kafka-console-consumer.sh --bootstrap-server my.broker.address:9092 --topic __debezium-heartbeat.postgres --from-beginning --consumer.config=/etc/kafka/consumer.properties
It would dump out all the heartbeat messages, and then show a new one every 1000ms.
但是,插槽的大小仍在增长.如果我执行诸如在表中插入虚拟记录之类的操作,则会将插槽设置为较小的滞后,这样就可以正常工作.
However, the size of the slots still grows and grows. If I do something like insert a dummy record into the table, it sets the slot back to a small lag, so that works.
尽管如此,我还是想跳动一下.我不想插入定期消息,因为这听起来会增加复杂性.为什么心跳没有减小插槽大小?
I would like to do this with a heartbeat though. I do not want to insert periodic messages since it sounds like it would add complexity. Why is the heartbeat not reducing the slot size?
推荐答案
请查看您确实需要发出定期消息,但是现在有一个帮助- https://issues.redhat.com/browse/DBZ-1815
You really need to emit the periodic messages but there is a help for it now - https://issues.redhat.com/browse/DBZ-1815
这篇关于Debezium Postgres Kafka Connector心跳未提交LSN的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!