Kafka-Connect vs Filebeat& Logstash [英] Kafka-Connect vs Filebeat & Logstash

问题描述

我正在寻找从卡夫卡消费，并将数据保存到Hadoop和Elasticsearch。
我目前看到有两种方法：使用Filebeat从Kafka消费并将其发送到ES并使用Kafka-Connect框架。有一个Kafka-Connect-HDFS和Kafka-Connect-Elasticsearch模块。

我不知道使用哪一个来发送流数据。虽然我认为，如果我想在某个时候从卡夫卡获取数据并将其放入Cassandra，我可以使用Kafka-Connect模块，但Filebeat不存在这样的功能。

解决方案

Kafka Connect可以处理流数据，更灵活一些。如果您只是要弹性，Filebeat是日志源的干净整合。然而，如果你从卡夫卡去一些不同的水槽，卡夫卡连接可能是你想要的。我建议您检查连接器集线器，以查看您现在可以使用的开源连接器的一些示例。 http：/ /www.confluent.io/product/connectors/

I'm looking to consume from Kafka and save data into Hadoop and Elasticsearch. I've seen 2 ways of doing this currently: using Filebeat to consume from Kafka and send it to ES and using Kafka-Connect framework. There is a Kafka-Connect-HDFS and Kafka-Connect-Elasticsearch module.

I'm not sure which one to use to send streaming data. Though I think that if I want at some point to take data from Kafka and place it into Cassandra I can use a Kafka-Connect module for that but no such feature exists for Filebeat.

解决方案

Kafka Connect can handle streaming data and is a bit more flexible. If you are just going to elastic, Filebeat is a clean integration for log sources. However, if you are going from Kafka to a number of different sinks, Kafka Connect is probably what you want. I'd recommend checking out the connector hub to see some examples of open source connectors at your disposal currently http://www.confluent.io/product/connectors/

这篇关于Kafka-Connect vs Filebeat& Logstash的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！