Kafka JDBC Sink连接器中的批次大小 [英] Batch Size in kafka jdbc sink connector

问题描述

我想通过jdbc接收器批量读取5000条记录，为此我在jdbc接收器配置文件中使用了batch.size：

I want to read only 5000 records in a batch through jdbc sink, for which I've used the batch.size in the jdbc sink config file:

name=jdbc-sink
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
tasks.max=1
batch.size=5000
topics=postgres_users

connection.url=jdbc:postgresql://localhost:34771/postgres?user=foo&password=bar
file=test.sink.txt
auto.create=true

但是由于将记录插入到记录中，batch.size无效。新记录插入到源数据库中时，数据库。

But the batch.size has no effect as records are getting inserted into the database when new records are inserted into the source database.

如何实现批量插入5000条记录？

How can I achieve to insert in a batch of 5000?

推荐答案

尚无直接解决方案来批量下沉记录，但是如果可行，我们会在属性下面给出try tune。我从没尝试过，但我对Kafka Sink Connector的理解只是让消费者消费消息主题。

There is no direct solution to sink records in batches but we give try tune below property if it works. I have never tried but my understanding Kafka Sink Connector nothing but consumer to consume message fron topic.

max.poll.records：一次调用poll（）返回的记录数

max.poll.records: The maximum number of records returned in a single call to poll()

consumer.fetch.min.bytes ：服务器应返回的最小数据量获取请求。如果没有足够的数据，请求将在等待请求之前等待大量数据积累

consumer.fetch.min.bytes: The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request

fetch.wait.max.ms：经纪人将在发送响应到消费者客户端之前等待这段时间，除非它有足够的数据来填充响应（fetch.message.max.bytes）

fetch.wait.max.ms: The broker will wait for this amount of time BEFORE sending a response to the consumer client, unless it has enough data to fill the response (fetch.message.max.bytes)

fetch.min.bytes：：代理在将响应发送给消费者客户端之前，代理将等待该数据量填满。

fetch.min.bytes: The broker will wait for this amount of data to fill BEFORE it sends the response to the consumer client.

这篇关于Kafka JDBC Sink连接器中的批次大小的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！