Kafka 未将文件更改发布到主题 [英] Kafka not publishing file changes to topic

问题描述

阅读:Kafka Connect FileStreamSource 忽略附加行

来自 2018 年的回答:

An Answer from 2018 states :

Kafka Connect does not "watch" or "tail" a file. I don't believe it is documented anywhere that it does do that.

现在看来 Kafka 确实支持这种阅读方式:

It seems Kafka does now support this as reading:

https://docs.confluent.io/5.5.0/connect/managing/configuring.html#standalone-example

确实说明文件已被监视:

does state the file is watched:

FileSource 连接器 FileSource 连接器从文件中读取数据并将其发送到 Apache Kafka®.超越所有人通用的配置连接器它只需要一个输入文件和输出主题作为属性.这是一个示例配置:

FileSource Connector The FileSource Connector reads data from a file and sends it to Apache Kafka®. Beyond the configurations common to all connectors it takes only an input file and output topic as properties. Here is an example configuration:

name=local-file-source connector.class=FileStreamSource tasks.max=1file=/tmp/test.txt topic=connect-test 此连接器将只读一个文件并将该文件中的数据发送到 Kafka.然后它会仅查看文件以获取附加更新.文件的任何修改已经发送到 Kafka 的行将不会被重新处理.

name=local-file-source connector.class=FileStreamSource tasks.max=1 file=/tmp/test.txt topic=connect-test This connector will read only one file and send the data within that file to Kafka. It will then watch the file for appended updates only. Any modification of file lines already sent to Kafka will not be reprocessed.

我的配置与发布的问题相同 Kafka Connect FileStreamSource 忽略附加行

My configuration is same as question posted Kafka Connect FileStreamSource ignores appended lines

connect-file-source.properties 包含:

connect-file-source.properties contains:

name=my-file-connector
connector.class=FileStreamSource
tasks.max=1
file=/data/users/zamara/suivi_prod/app/data/logs.txt
topic=connect-test

开始独立连接

connect-standalone connect-standalone.properties connect-file-source.properties 

将文件 logs.txt 的所有内容添加到主题 connect-test 中，在 logs.txt 中添加新行不会将行添加到主题中.是否需要配置才能让 Kafka 监视文件，以便将添加到 logs.txt 的新数据添加到主题 connect-test 中?

Adds all the contents of the file logs.txt to the topic connect-test , adding new lines to logs.txt does not add the lines to the topic. Is there configuration required to enable Kafka to watch the file so that new data added to logs.txt is added to the topic connect-test ?

推荐答案

除非您只是为了教育目的而尝试使用 FileStreamSource，否则您将陷入死胡同.连接器仅作为示例连接器存在.

Unless you're just experimenting with FileStreamSource for educational purposes, you're heading down a blind alley here. The connector only exists as a sample connector.

要将文件摄取到 Kafka 中，请使用 Kafka Connect Spooldir、Kafka Connect FilePulse，或查看 Elastic 的 Filebeat 之类的东西.

To ingest files into Kafka use Kafka Connect Spooldir, Kafka Connect FilePulse, or look at things like Filebeat from Elastic.

这篇关于Kafka 未将文件更改发布到主题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！