通过运行 Docker-Compose 和 Spring Boot 的 IP 地址访问远程主机中的 Kafka [英] Access Kafka in Remote Host by IP Address running with Docker-Compose and Spring Boot

查看：26
本文介绍了通过运行 Docker-Compose 和 Spring Boot 的 IP 地址访问远程主机中的 Kafka的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我有这个 docker-compose.yml，我在其中运行 Zookeeper、Kafka、Kafka Connect 和 KafDrop，问题是，当我在本地运行时，我可以从我的 Spring Boot 应用程序连接以使用一些主题消息.

I have this docker-compose.yml in which I run Zookeeper, Kafka, Kafka Connect, and KafDrop, the thing is, when I run locally I can connect from my Spring Boot application to consume some topic messages.

我需要的是在 Linux 机器上运行相同的配置，并且能够以相同的方式从 Spring Boot 应用程序连接.

What I need is to run the same configuration on a Linux machine and be able to connect from the Spring Boot application the same way.

当在 Linux 机器上远程运行它时，一切似乎都运行正常，但是当我尝试从 Spring Boot 应用程序连接时，我收到一些错误，表明连接中出现了错误.

When run it remotely on the Linux machine everything seems to be running Ok, but when I try to connect from the Spring Boot application I receive some erros showing that somethin is wrong in the connection.

我会试着一步一步地解释，看看是否有人可以给一个光"对此:

I will try to explain step by step and see if someone can give a "light" on that:

docker-compose.yml:

docker-compose.yml:

version: '3'

services:

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    networks: 
      - broker-kafka
    ports:
      - 2181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    networks: 
      - broker-kafka
    restart: unless-stopped
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: 
         INTERNAL://kafka:29092,
         EXTERNAL://localhost:9092
      KAFKA_ADVERTISED_LISTENERS: 
         INTERNAL://kafka:29092,
         EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 
         INTERNAL:PLAINTEXT,
         EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_RETENTION_HOURS: 12
    
  connect:
    image: cdc:latest
    networks: 
      - broker-kafka
    depends_on:
      - zookeeper
      - kafka
    ports:
      - 8083:8083
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka:29092
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: connect-1
      CONNECT_CONFIG_STORAGE_TOPIC: connect-1-config
      CONNECT_OFFSET_STORAGE_TOPIC: connect-1-offsets
      CONNECT_STATUS_STORAGE_TOPIC: connect-1-status
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_OFFSET.STORAGE.REPLICATION.FACTOR: 1
      CONNECT_CONFIG.STORAGE.REPLICATION.FACTOR: 1
      CONNECT_OFFSET.STORAGE.PARTITIONS: 1
      CONNECT_STATUS.STORAGE.REPLICATION.FACTOR: 1
      CONNECT_STATUS.STORAGE.PARTITIONS: 1
      CONNECT_REST_ADVERTISED_HOST_NAME: localhost
      
  kafdrop:
    image: obsidiandynamics/kafdrop:latest
    networks: 
      - broker-kafka
    depends_on:
      - kafka
    ports:
      - 19000:9000
    environment:
      KAFKA_BROKERCONNECT: kafka:29092
      
networks: 
  broker-kafka:
    driver: bridge

我需要的是将这台 IP 机器暴露给我的网络，以便我的 Spring Boot 应用程序访问.假设这台 Linux 机器的 IP 为 10.12.54.99.如何通过以下方式访问 Kafka:10.12.54.99:9090?

What I need is to expose to my network this IP machine to be accessed by my Spring Boot application. Let´s suppose this Linux machine has the IP 10.12.54.99. How can I make it Kafka be accessible by: 10.12.54.99:9090 ?

这是我的 application.properties:

Here is my application.properties:

spring.kafka.bootstrap-servers=10.12.54.99:9092

spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer

spring.kafka.consumer.enable-auto-commit=false
spring.kafka.consumer.auto-commit-interval=100
spring.kafka.consumer.max-poll-records=10
spring.kafka.consumer.key-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
spring.kafka.consumer.group-id=connect-sql-server
spring.kafka.consumer.auto-offset-reset=earliest

spring.kafka.listener.ack-mode=manual-immediate
spring.kafka.listener.poll-timeout=3000
spring.kafka.listener.concurrency=3

spring.kafka.properties.spring.deserializer.key.delegate.class=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.properties.spring.deserializer.value.delegate.class=org.apache.kafka.common.serialization.StringDeserializer

这是唯一一个特定于消费者的应用程序(此处未使用生产者).

This is a only consumer-specif application (no producers are used here).

当我运行应用程序时:

2020-12-07 10:59:40.361  WARN 58716 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-connect-sql-server-1, groupId=connect-sql-server] Connection to node -1 (/10.12.54.99:9092) could not be established. Broker may not be available.
2020-12-07 10:59:40.362  WARN 58716 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-connect-sql-server-1, groupId=connect-sql-server] Bootstrap broker 10.12.54.99:9092 (id: -1 rack: null) disconnected

在 Linux 防火墙机器中启用了所有防火墙端口.

All the firewall ports are enabled in the Linux firewall machie.

非常感谢任何启发.

推荐答案

您需要绑定服务器的公共 ip 才能远程访问 broker.但是，如果您不想对 ip 进行硬编码，则可以使用 .env 文件.

You need to bind your server's public ip in order to be able to access brokers remotely. However if you don't want to hardcode the ip, you can use .env file.

执行以下操作:

  1. 创建 config.env 文件.

  1. Create config.env file.

在 config.env 中添加这一行并添加您的主机 ip，如下所示:

Add this line in config.env and add your host ip as below:

DOCKER_HOST_IP=111.111.11.111

DOCKER_HOST_IP=111.111.11.111

更新您的 docker-compose:

Update your docker-compose:

version: '3'

services:

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    networks: 
      - broker-kafka
    ports:
      - ${DOCKER_HOST_IP:-127.0.0.1}:2181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    networks: 
      - broker-kafka
    restart: unless-stopped
    depends_on:
      - zookeeper
    ports:
      - ${DOCKER_HOST_IP:-127.0.0.1}:9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: 
         INTERNAL://kafka:29092,
         EXTERNAL://localhost:9092
      KAFKA_ADVERTISED_LISTENERS: 
         INTERNAL://kafka:29092,
         EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 
         INTERNAL:PLAINTEXT,
         EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_RETENTION_HOURS: 12
    
  connect:
    image: cdc:latest
    networks: 
      - broker-kafka
    depends_on:
      - zookeeper
      - kafka
    ports:
      - 8083:8083
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka:29092
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: connect-1
      CONNECT_CONFIG_STORAGE_TOPIC: connect-1-config
      CONNECT_OFFSET_STORAGE_TOPIC: connect-1-offsets
      CONNECT_STATUS_STORAGE_TOPIC: connect-1-status
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_OFFSET.STORAGE.REPLICATION.FACTOR: 1
      CONNECT_CONFIG.STORAGE.REPLICATION.FACTOR: 1
      CONNECT_OFFSET.STORAGE.PARTITIONS: 1
      CONNECT_STATUS.STORAGE.REPLICATION.FACTOR: 1
      CONNECT_STATUS.STORAGE.PARTITIONS: 1
      CONNECT_REST_ADVERTISED_HOST_NAME: localhost
      
  kafdrop:
    image: obsidiandynamics/kafdrop:latest
    networks: 
      - broker-kafka
    depends_on:
      - kafka
    ports:
      - 19000:9000
    environment:
      KAFKA_BROKERCONNECT: kafka:29092
      
networks: 
  broker-kafka:
    driver: bridge

如果没有找到 DOCKER_HOST_IP，它将绑定到 127.0.0.1.

It will bind to 127.0.0.1, if DOCKER_HOST_IP is not found.

  1. 运行以下命令:

sudo docker-compose -f path-to-docker-compose.yml --env-file path-to-config.env up -d --force-recreate

这篇关于通过运行 Docker-Compose 和 Spring Boot 的 IP 地址访问远程主机中的 Kafka的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆