如何使用 haproxy 对 Kafka Boostrap 进行负载平衡? [英] How to load balance Kafka Boostrap with haproxy?

问题描述

我有一个由 3 台在 AWS 上运行的机器组成的 kafka 集群:卡夫卡1到卡夫卡3

I have a kafka cluster consisting of 3 machines running on AWS: kafka1 to kafka3

我正在使用新样式的 kafka 消费者 (>0.8).

I'm using the new style kafka consumers (>0.8).

我知道 kafka 客户端连接到其中一台 kafka 服务器，获取服务器元数据，然后直接连接到代理.

I know that the kafka client connects to one of the kafka servers, grabs server metadata and then connects to the brokers directly.

我想确保在代理失败的情况下，客户端仍然能够获取元数据.

I want to ensure that in the case of a broker failure that the clients are still able to grab metadata.

为此，我有一个具有以下配置的 HAProxy 负载均衡器:

For this I have an HAProxy load balancer with the following config:

listen kafka
      bind *:9092
      mode tcp
      balance roundrobin
      no option clitcpka
      option forceclose
      timeout check 5s
      server kafka1 kafka1.example.com:9092 check inter 3s fastinter 1s
      server kafka2 kafka2.example.com:9092 check inter 3s fastinter 1s
      server kafka3 kafka3.example.com:9092 check inter 3s fastinter 1s

这个想法是，如果其中一个代理出现故障，它将从轮换中删除，并从其他代理中获取元数据.它还允许我透明地向集群添加更多代理.

The idea is that if one of the brokers goes down it is removed from rotation and metadata is grabbed from one of the others. It also allows me to add more brokers to the cluster transparently.

但是，这会导致我的 kafka 客户端出现问题.PipelineDB 直接拒绝从主题中消费，Python 的融合 kafka 库也是如此.Kafkacat 消费，但过一段时间开始报错:

However, this is causing issues with my kafka clients. PipelineDB straight up refuses to consume from topics, so does the confluent kafka library for Python. Kafkacat consumes, but starts giving errors after a while:

% ERROR: Local: Broker transport failure: kafka.example.com:9092/bootstrap: Receive failed: Disconnected
% ERROR: Local: Broker transport failure: kafka.example.com:9092/bootstrap: Connection closed

我在网上找不到关于如何对 Kafka 引导程序进行负载平衡的任何信息.另一种方法是只配置一个包含多个 A 记录的 DNS 条目，但我遇到了其中一个代理宕机的问题.

I can't find any info online on how to load balance the Kafka bootstrap procedure. An alternative would be to just configure a DNS entry with multiple A records, but I run into the problem of one of the brokers being down.

推荐答案

只需将所有三个引导服务器指定为逗号分隔列表.然后，如果一个发生故障，它只会查询下一个元数据.

Just specify all three bootstrap servers as a comma separated list. Then if one goes down, it will just query the next one for meta-data.

这篇关于如何使用 haproxy 对 Kafka Boostrap 进行负载平衡?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！