使用 docker-compose 设置 Kafka [英] Kafka setup with docker-compose

问题描述

我目前正在使用 Docker 设置 Kafka.我已经设法使用已发布的融合图像设置 Zookeeper 和 Kafka，请参阅以下 docker-compose 文件:

Hi I'm currently setting up Kafka with Docker. I've managed to setup Zookeeper and Kafka with the published confluent image, see following docker-compose file:

version: '2'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:3.2.0
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    restart: always

  kafka:
    image: confluentinc/cp-kafka:3.2.0
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.99.100:9092
      LISTENERS: PLAINTEXT://0.0.0.0:9092
    restart: always

  kafka-rest:
   image: confluentinc/cp-kafka-rest:3.2.0
   container_name: kafka-rest
   depends_on:
     - kafka
   ports:
     - '8082:8082'
   environment:
     KAFKA_REST_ZOOKEEPER_CONNECT: 'zookeeper:2181'
     KAFKA_REST_LISTENERS: http://kafka-rest:8082
     KAFKA_REST_SCHEMA_REGISTRY_URL: http://schema-registry:8081
     KAFKA_REST_HOST_NAME: kafka-rest
   restart: always

 schema-registry:
   image: confluentinc/cp-schema-registry:3.2.0
   container_name: schema-registry
   depends_on:
     - kafka
   ports:
     - '8081'
   environment:
     SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181'
     SCHEMA_REGISTRY_HOST_NAME: schema-registry
     SCHEMA_REGISTRY_LISTENERS: http://schema-registry:8081
   restart: always

 connect:
   image: confluentinc/cp-kafka-connect:3.2.0
   container_name: kafka-connect
   depends_on:
     - zookeeper
     - kafka
     - schema-registry
   ports:
     - "8083:8083"
   restart: always
   environment:
     CONNECT_BOOTSTRAP_SERVERS: 'kafka:9092'
     CONNECT_REST_ADVERTISED_HOST_NAME: connect
     CONNECT_REST_PORT: 8083
     CONNECT_GROUP_ID: compose-connect-group
     CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
     CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
     CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
     CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
     CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
     CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
     CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
     CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
     CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
     CONNECT_ZOOKEEPER_CONNECT: "zookeeper:2181"

现在我通过正确地将adverted.listener 属性设置为PLAINTEXT://{DOCKER_MACHINE_IP}:9092，成功地将Kafka 容器正确地暴露给我的非dockerized 应用程序，但正如你所看到的，我还添加了其他融合应用程序来扩展我的 Kafka 设置(Kafka REST、Schema-Registry).由于adverted.listener 属性，这些无法再连接到我的Kafka 实例.

Now I've managed to expose correctly the Kafka container to my non-dockerized applications by correctly setting the advertised.listener property to PLAINTEXT://{DOCKER_MACHINE_IP}:9092, but as you can see I've also added other confluent applications to extend my Kafka setup (Kafka REST, Schema-Registry). These can no longer connect to my Kafka instance because of the advertised.listener property.

我可以将其更改为正确的容器主机名 --> PLAINTEXT://kafka:9092 但随后我无法使用其他应用程序再次访问 kafka 实例.有什么简单的方法可以解决这个问题吗?

I could change it to the correct container hostname --> PLAINTEXT://kafka:9092 but then I lose the ability to reach the kafka instance once again with my other apps. Is there any easy way to solve this issue?

推荐答案

Omar，也许您已经解决了您的问题，但为了将来参考，Hans Jespersen 的评论对我有用，即使在 Windows 上也是如此.

Omar, maybe you've already resolved your problem, but for future reference, Hans Jespersen's comment did the trick for me, even on Windows.

以管理员身份打开 C:\Windows\System32\drivers\etc\hosts 并添加以下行以将 kafka 代理公开为 localhost.<代码>127.0.0.1 经纪人

As admin, open C:\Windows\System32\drivers\etc\hosts and add the following line to expose the kafka broker as localhost. 127.0.0.1 broker

我的docker-compose.yml文件如下所示:

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper
    hostname: zookeeper
    extra_hosts:
    - "moby:127.0.0.1"
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka
    hostname: broker
    extra_hosts:
    - "moby:127.0.0.1"
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:9092'
      KAFKA_DEFAULT_REPLICATION_FACTOR: 1

  schema_registry:
    image: confluentinc/cp-schema-registry
    hostname: schema_registry
    # extra_hosts:
    # - "moby:127.0.0.1"
    depends_on:
      - zookeeper
      - broker
    ports:
      - '8081:8081'
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema_registry
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181'

  kafka-rest:
    image: confluentinc/cp-kafka-rest
    container_name: kafka-rest
    extra_hosts:
    - "moby:127.0.0.1"
    depends_on:
      - zookeeper
      - broker
    ports:
      - '8082:8082'
    environment:
      KAFKA_REST_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_REST_LISTENERS: http://kafka-rest:8082
      KAFKA_REST_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      KAFKA_REST_HOST_NAME: kafka-rest

或者，公开我的笔记本电脑的当前 IP 地址(使用 ipconfig/all)也可以，但这样做的缺点是，每当我的网络发生变化时，我都必须更改 docker-compose.yml文件也是.

Alternatively, exposing my laptop's current IP address (using ipconfig /all) works too, but this has the disadvantage that, whenever my network changes, I would have to change the docker-compose.yml file too.

这篇关于使用 docker-compose 设置 Kafka的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！