用于 Azure Blob 存储的 Kafka 连接器 [英] Kafka Connector for Azure Blob Storage

问题描述

我需要将推送到 Kafka 的消息存储在深度存储中.我们正在使用 Azure 云服务，所以我认为 Azure Blob 存储可能是更好的选择.我想使用 Kafka Connect 的接收器连接器 API 将数据推送到 Azure Blob.Kafka 文档主要建议 HDFS 导出数据，但是，在这种情况下，我需要一个运行 Hadoop 的 Linux VM，我猜这会很昂贵.我的问题是 Azure Blob 存储是存储 JSON 对象的合适选择，而构建自定义接收器连接器是这种情况下的合理解决方案吗?

I need to store the messages pushed to Kafka in a deep storage. We are using Azure cloud services so I suppose Azure Blob storage could be a better option. I want to use Kafka Connect's sink connector API to push data to Azure Blob. Kafka documentation mostly suggests HDFS to export data however, in that case I need a Linux VM running Hadoop that will be costly I guess. My question is Azure Blob storage an appropriate choice to store JSON objects and building a custom sink connector is a reasonable solution for this case?

推荐答案

自定义接收器连接器绝对有效.Kafka Connect 是绝对设计的，因此您可以插入连接器.事实上，连接器的开发是完全联合的.Confluent 的 JDBC 和 HDFS 连接器最初的实现仅仅是因为这两个用例的流行，但还有更多(我们保留了一个我们知道的连接器列表 这里.

A custom sink connector definitely works. Kafka Connect was absolutely designed so you could plugin connectors. In fact, connector development is entirely federated. Confluent's JDBC and HDFS connectors were implemented first simply due to the popularity of those two use cases, but there are many more (we keep a list of connectors we're aware of here.

就 Azure blob 存储是否合适而言，您提到了 JSON 对象.我认为您唯一需要考虑的是对象的大小以及 Azure 存储是否可以处理大小和大小.对象的数量很好.我不确定 Azure 存储的特性，但在许多其他对象存储系统中，您可能需要将许多对象聚合到一个 blob 中才能为大量对象获得良好的性能(即您可能需要一种支持许多 JSON 对象的文件格式).

In terms of whether Azure blob storage is appropriate, you mention JSON objects. I think the only thing you'll want to consider is the size of the objects and whether Azure storage will handle the size & number of objects well. I am not sure about Azure storage's characteristics, but in many other object storage systems you might need to aggregate many objects into a single blob to get good performance for a large number of objects (i.e. you might need a file format that supports many JSON objects).

这篇关于用于 Azure Blob 存储的 Kafka 连接器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！