如何使用haproxy负载均衡Kafka Boostrap? [英] How to load balance Kafka Boostrap with haproxy?

问题描述

我有一个由3台在AWS上运行的机器组成的kafka集群: kafka1至kafka3

I have a kafka cluster consisting of 3 machines running on AWS: kafka1 to kafka3

我正在使用新型kafka消费者(> 0.8).

I'm using the new style kafka consumers (>0.8).

我知道kafka客户端连接到其中一台kafka服务器，获取服务器元数据，然后直接连接到代理.

I know that the kafka client connects to one of the kafka servers, grabs server metadata and then connects to the brokers directly.

我想确保在代理失败的情况下，客户端仍然能够获取元数据.

I want to ensure that in the case of a broker failure that the clients are still able to grab metadata.

为此，我有一个具有以下配置的HAProxy负载均衡器:

For this I have an HAProxy load balancer with the following config:

listen kafka
      bind *:9092
      mode tcp
      balance roundrobin
      no option clitcpka
      option forceclose
      timeout check 5s
      server kafka1 kafka1.example.com:9092 check inter 3s fastinter 1s
      server kafka2 kafka2.example.com:9092 check inter 3s fastinter 1s
      server kafka3 kafka3.example.com:9092 check inter 3s fastinter 1s

这个想法是，如果其中一个代理发生故障，则将其从轮换中删除，并将元数据从另一个代理中获取.它还使我可以透明地向群集中添加更多代理.

The idea is that if one of the brokers goes down it is removed from rotation and metadata is grabbed from one of the others. It also allows me to add more brokers to the cluster transparently.

但是，这导致我的kafka客户出现问题. PipelineDB直接拒绝从主题中消费，因此用于Python的融合kafka库也是如此. Kafkacat消耗掉了，但过了一会儿开始出错:

However, this is causing issues with my kafka clients. PipelineDB straight up refuses to consume from topics, so does the confluent kafka library for Python. Kafkacat consumes, but starts giving errors after a while:

% ERROR: Local: Broker transport failure: kafka.example.com:9092/bootstrap: Receive failed: Disconnected
% ERROR: Local: Broker transport failure: kafka.example.com:9092/bootstrap: Connection closed

我在网上找不到任何有关如何平衡Kafka引导程序负载的信息.一种替代方法是只配置具有多个A记录的DNS条目，但是我遇到了其中一个代理关闭的问题.

I can't find any info online on how to load balance the Kafka bootstrap procedure. An alternative would be to just configure a DNS entry with multiple A records, but I run into the problem of one of the brokers being down.

推荐答案

只需将所有三个引导服务器指定为以逗号分隔的列表即可.然后，如果其中一个出现故障，它将仅查询下一个以获取元数据.

Just specify all three bootstrap servers as a comma separated list. Then if one goes down, it will just query the next one for meta-data.

这篇关于如何使用haproxy负载均衡Kafka Boostrap?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！