Kafka未发布对主题的文件更改 [英] Kafka not publishing file changes to topic

问题描述

阅读: Kafka Connect FileStreamSource会忽略附加的行

2018年的答案:

Kafka Connect does not "watch" or "tail" a file. I don't believe it is documented anywhere that it does do that.

似乎卡夫卡现在支持阅读:

It seems Kafka does now support this as reading:

https://docs.confluent.io/5.5.0/connect/managing/configuring.html#standalone-example

说明正在监视文件:

FileSource连接器FileSource连接器从文件读取数据并将其发送到ApacheKafka®.超越所有人共同的配置连接器仅将输入文件和输出主题作为属性.这是一个示例配置:

FileSource Connector The FileSource Connector reads data from a file and sends it to Apache Kafka®. Beyond the configurations common to all connectors it takes only an input file and output topic as properties. Here is an example configuration:

name =本地文件源连接器.class = FileStreamSourcetasks.max = 1file =/tmp/test.txt topic = connect-test此连接器将只读一个文件，然后将该文件中的数据发送给Kafka.然后它将监视文件中仅附加的更新.文件的任何修改已经发送到Kafka的行将不会被重新处理.

name=local-file-source connector.class=FileStreamSource tasks.max=1 file=/tmp/test.txt topic=connect-test This connector will read only one file and send the data within that file to Kafka. It will then watch the file for appended updates only. Any modification of file lines already sent to Kafka will not be reprocessed.

我的配置与发布的问题相同 Kafka Connect FileStreamSource会忽略附加行

My configuration is same as question posted Kafka Connect FileStreamSource ignores appended lines

connect-file-source.properties包含:

connect-file-source.properties contains:

name=my-file-connector
connector.class=FileStreamSource
tasks.max=1
file=/data/users/zamara/suivi_prod/app/data/logs.txt
topic=connect-test

开始与之独立连接

connect-standalone connect-standalone.properties connect-file-source.properties 

将文件 logs.txt 的所有内容添加到主题 connect-test 中，不将新行添加到 logs.txt 将行添加到主题.是否需要进行配置才能使Kafka监视文件，以便将添加到 logs.txt 的新数据添加到主题 connect-test ?

Adds all the contents of the file logs.txt to the topic connect-test , adding new lines to logs.txt does not add the lines to the topic. Is there configuration required to enable Kafka to watch the file so that new data added to logs.txt is added to the topic connect-test ?

推荐答案

除非您出于教育目的而尝试使用 FileStreamSource ，否则您将在此处进入盲区.该连接器仅作为示例连接器存在.

Unless you're just experimenting with FileStreamSource for educational purposes, you're heading down a blind alley here. The connector only exists as a sample connector.

要将文件提取到Kafka中，请使用Kafka Connect Spooldir，Kafka Connect FilePulse或查看Elastic中的Filebeat之类的东西.

To ingest files into Kafka use Kafka Connect Spooldir, Kafka Connect FilePulse, or look at things like Filebeat from Elastic.

这篇关于Kafka未发布对主题的文件更改的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！