用docker-compose设置Kafka [英] Kafka setup with docker-compose

问题描述

我目前正在使用Docker设置Kafka。我已经成功地用发布的融合图像设置了Zookeeper和Kafka，请参阅以下docker-compose文件：

Hi I'm currently setting up Kafka with Docker. I've managed to setup Zookeeper and Kafka with the published confluent image, see following docker-compose file:

version: '2'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:3.2.0
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    restart: always

  kafka:
    image: confluentinc/cp-kafka:3.2.0
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.99.100:9092
      LISTENERS: PLAINTEXT://0.0.0.0:9092
    restart: always

  kafka-rest:
   image: confluentinc/cp-kafka-rest:3.2.0
   container_name: kafka-rest
   depends_on:
     - kafka
   ports:
     - '8082:8082'
   environment:
     KAFKA_REST_ZOOKEEPER_CONNECT: 'zookeeper:2181'
     KAFKA_REST_LISTENERS: http://kafka-rest:8082
     KAFKA_REST_SCHEMA_REGISTRY_URL: http://schema-registry:8081
     KAFKA_REST_HOST_NAME: kafka-rest
   restart: always

 schema-registry:
   image: confluentinc/cp-schema-registry:3.2.0
   container_name: schema-registry
   depends_on:
     - kafka
   ports:
     - '8081'
   environment:
     SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181'
     SCHEMA_REGISTRY_HOST_NAME: schema-registry
     SCHEMA_REGISTRY_LISTENERS: http://schema-registry:8081
   restart: always

 connect:
   image: confluentinc/cp-kafka-connect:3.2.0
   container_name: kafka-connect
   depends_on:
     - zookeeper
     - kafka
     - schema-registry
   ports:
     - "8083:8083"
   restart: always
   environment:
     CONNECT_BOOTSTRAP_SERVERS: 'kafka:9092'
     CONNECT_REST_ADVERTISED_HOST_NAME: connect
     CONNECT_REST_PORT: 8083
     CONNECT_GROUP_ID: compose-connect-group
     CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
     CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
     CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
     CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
     CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
     CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
     CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
     CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
     CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
     CONNECT_ZOOKEEPER_CONNECT: "zookeeper:2181"

现在，我已经通过正确设置了advertised.listener属性设置为PLAINTEXT：// {DOCKER_MACHINE_IP}：9092，但是如您所见，我还添加了其他融合应用程序来扩展我的Kafka设置（Kafka REST，架构注册表）。由于advertised.listener属性，它们无法再连接到我的Kafka实例。

Now I've managed to expose correctly the Kafka container to my non-dockerized applications by correctly setting the advertised.listener property to PLAINTEXT://{DOCKER_MACHINE_IP}:9092, but as you can see I've also added other confluent applications to extend my Kafka setup (Kafka REST, Schema-Registry). These can no longer connect to my Kafka instance because of the advertised.listener property.

我可以将其更改为正确的容器主机名-> PLAINTEXT：// kafka：9092，但随后我失去了再次访问我的kafka实例的能力应用。有没有简单的方法可以解决此问题？

I could change it to the correct container hostname --> PLAINTEXT://kafka:9092 but then I lose the ability to reach the kafka instance once again with my other apps. Is there any easy way to solve this issue?

推荐答案

Omar，也许您已经解决了问题，但供将来参考，即使在Windows上，Hans Jespersen的评论也对我有用。

Omar, maybe you've already resolved your problem, but for future reference, Hans Jespersen's comment did the trick for me, even on Windows.

以管理员身份打开 C：\Windows\System32\drivers\etc\hosts ，然后添加以下行以将kafka代理公开为localhost。

127.0.0.1经纪人


As admin, open C:\Windows\System32\drivers\etc\hosts and add the following line to expose the kafka broker as localhost. 127.0.0.1 broker

而我的 docker-compose.yml 文件如下所示：

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper
    hostname: zookeeper
    extra_hosts:
    - "moby:127.0.0.1"
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka
    hostname: broker
    extra_hosts:
    - "moby:127.0.0.1"
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:9092'
      KAFKA_DEFAULT_REPLICATION_FACTOR: 1

  schema_registry:
    image: confluentinc/cp-schema-registry
    hostname: schema_registry
    # extra_hosts:
    # - "moby:127.0.0.1"
    depends_on:
      - zookeeper
      - broker
    ports:
      - '8081:8081'
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema_registry
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181'

  kafka-rest:
    image: confluentinc/cp-kafka-rest
    container_name: kafka-rest
    extra_hosts:
    - "moby:127.0.0.1"
    depends_on:
      - zookeeper
      - broker
    ports:
      - '8082:8082'
    environment:
      KAFKA_REST_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_REST_LISTENERS: http://kafka-rest:8082
      KAFKA_REST_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      KAFKA_REST_HOST_NAME: kafka-rest

或者，公开笔记本电脑的当前IP地址（使用ipconfig / all）也可以，但是这样做的缺点是每次网络更改时，我也必须更改 docker-compose.yml 文件。

Alternatively, exposing my laptop's current IP address (using ipconfig /all) works too, but this has the disadvantage that, whenever my network changes, I would have to change the docker-compose.yml file too.

这篇关于用docker-compose设置Kafka的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！