Kafka 连接教程停止工作 [英] Kafka connect tutorial stopped working

问题描述

我在此链接中遵循第 7 步(使用 Kafka Connect 导入/导出数据):

http://kafka.apache.org/documentation.html#quickstart

它运行良好，直到我删除了test.txt"文件.主要是因为这就是 log4j 文件的工作方式.一段时间后，文件将被旋转 - 我的意思是 - 它将被重命名 &将开始写入具有相同名称的新文件.

但是之后，我删除了test.txt"，连接器停止工作.我重新启动了连接器、代理、zookeeper 等，但是test.txt"中的新行不会进入connect-test"主题 &因此不会转到test.sink.txt"文件.

我该如何解决这个问题?

解决方案

连接器保留其从文件中读取的最后位置"的标签，因此万一它在读取文件时崩溃，它可以从中断处继续.

问题是您删除了文件而没有将偏移量重置为 0，因此它基本上看不到任何新数据，因为它等待新数据从头开始以特定字符数开始显示...

重置偏移的解决方法.如果您在独立模式下使用连接，默认情况下偏移量存储在/tmp/connect.offsets 中，只需从那里删除它们即可.

从长远来看，我们需要一个更好的文件连接器:)

I was following step #7 (Use Kafka Connect to import/export data) at this link:

http://kafka.apache.org/documentation.html#quickstart

It was working well until I deleted the 'test.txt' file. Mainly because that's how log4j files would work. After certain time, the file will get rotated - I mean - it will be renamed & a new file with the same name will start getting written to.

But after, I deleted 'test.txt', the connector stopped working. I restarted connector, broker, zookeeper etc, but the new lines from 'test.txt' are not going to the 'connect-test' topic & therefore are not going to the 'test.sink.txt' file.

How can I fix this?

解决方案

The connector keeps tabs of its "last location read from a file", so in case it crashes while reading the file, it can continue where it left off.

The problem is that you deleted the file without resetting the offsets to 0, so it basically doesn't see any new data since it waits for new data to show starting at a specific character count from the beginning...

The work-around if to reset the offsets. If you are using connect in stand-alone mode, the offsets are stored in /tmp/connect.offsets by default, just delete them from there.

In the long term, we need a better file connector :)

这篇关于Kafka 连接教程停止工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！