Kafka Connect教程停止工作 [英] Kafka connect tutorial stopped working

问题描述

我在此链接上执行了第7步(使用Kafka Connect导入/导出数据):

http://kafka.apache.org/documentation.html#quickstart

在删除"test.txt"文件之前，它一直运行良好.主要是因为这就是log4j文件的工作方式.一段时间后，文件将旋转-我的意思是-文件将被重命名&具有相同名称的新文件将开始被写入.

但是，之后，我删除了"test.txt"，连接器停止工作.我重新启动了连接器，代理，zookeeper等，但是来自"test.txt"的新行不会进入"connect-test"主题&因此不会转到"test.sink.txt"文件.

我该如何解决?

解决方案

连接器保留其从文件读取的最后位置"的选项卡，因此在读取文件时崩溃的情况下，连接器可以从中断处继续.

问题是您删除文件时未将偏移量重置为0，因此它基本上看不到任何新数据，因为它等待新数据从头开始显示特定的字符数...

解决方法，如果要重置偏移量.如果您以独立模式使用connect，则默认情况下，偏移量存储在/tmp/connect.offsets中，只需从那里删除它们即可.

从长远来看，我们需要一个更好的文件连接器:)

I was following step #7 (Use Kafka Connect to import/export data) at this link:

http://kafka.apache.org/documentation.html#quickstart

It was working well until I deleted the 'test.txt' file. Mainly because that's how log4j files would work. After certain time, the file will get rotated - I mean - it will be renamed & a new file with the same name will start getting written to.

But after, I deleted 'test.txt', the connector stopped working. I restarted connector, broker, zookeeper etc, but the new lines from 'test.txt' are not going to the 'connect-test' topic & therefore are not going to the 'test.sink.txt' file.

How can I fix this?

解决方案

The connector keeps tabs of its "last location read from a file", so in case it crashes while reading the file, it can continue where it left off.

The problem is that you deleted the file without resetting the offsets to 0, so it basically doesn't see any new data since it waits for new data to show starting at a specific character count from the beginning...

The work-around if to reset the offsets. If you are using connect in stand-alone mode, the offsets are stored in /tmp/connect.offsets by default, just delete them from there.

In the long term, we need a better file connector :)

这篇关于Kafka Connect教程停止工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！