kafka jdbc sink 连接器中的批量大小 [英] Batch Size in kafka jdbc sink connector

查看:32
本文介绍了kafka jdbc sink 连接器中的批量大小的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我想通过 jdbc sink 批量读取 5000 条记录,为此我使用了 jdbc sink 配置文件中的 batch.size:

I want to read only 5000 records in a batch through jdbc sink, for which I've used the batch.size in the jdbc sink config file:

name=jdbc-sink
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
tasks.max=1
batch.size=5000
topics=postgres_users

connection.url=jdbc:postgresql://localhost:34771/postgres?user=foo&password=bar
file=test.sink.txt
auto.create=true

但是当新记录插入到源数据库时,batch.size 没有影响,因为记录被插入到数据库中.

But the batch.size has no effect as records are getting inserted into the database when new records are inserted into the source database.

如何实现批量插入5000条?

How can I achieve to insert in a batch of 5000?

推荐答案

没有直接的解决方案来批量接收记录,但如果它有效,我们会在下面的属性中尝试调整.我从未尝试过,但我的理解 Kafka Sink Connector 只不过是一个消费者来消费来自主题的消息.

There is no direct solution to sink records in batches but we give try tune below property if it works. I have never tried but my understanding Kafka Sink Connector nothing but a consumer to consume message from the topic.

ma​​x.poll.records: 一次调用 poll() 时返回的最大记录数

max.poll.records: The maximum number of records returned in a single call to poll()

consumer.fetch.min.bytes:服务器应为获取请求返回的最小数据量.如果没有足够的数据可用,请求将在响应请求之前等待足够多的数据积累

consumer.fetch.min.bytes: The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request

fetch.wait.max.ms: 代理将在向消费者客户端发送响应之前等待这段时间,除非它有足够的数据来填充响应(fetch.message.max.bytes)

fetch.wait.max.ms: The broker will wait for this amount of time BEFORE sending a response to the consumer client unless it has enough data to fill the response (fetch.message.max.bytes)

fetch.min.bytes:代理将等待此数据量填满,然后再将响应发送给消费者客户端.

fetch.min.bytes: The broker will wait for this amount of data to fill BEFORE it sends the response to the consumer client.

这篇关于kafka jdbc sink 连接器中的批量大小的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆