如何在内存或嵌入式 kafka 中实现而不用于测试目的? [英] How do I implement in memory or embedded kafka not for testing purposes?

问题描述

我希望在仅允许 Kafka 由上述应用程序运行的环境中部署 Spring Boot 应用程序.我的应用程序将成为 Kafka 生产者和消费者.有没有办法在启动时运行内存中的实例，该实例可用于测试以外的目的?或者，有没有办法启动一个 Spring Boot 应用程序，如果它不能作为生产者和消费者连接到 Kafka 就不会失败?

I am looking to deploy a spring boot app in an environment that only allows Kafka to be run by the aforementioned app. My app will be a Kafka producer and consumer. Is there a way to run an in memory instance on startup that can be used for purposes other than testing? Alternatively, is there a way to startup a spring boot app that will not fail if it cannot connect to Kafka as a producer and consumer?

这是一个临时解决方案，直到我们能够在这种环境中部署 Kafka.该应用程序不会生成和使用自己的记录.它是多应用程序部署的一部分，其中每个应用程序都为其他应用程序生成并使用其他应用程序的 Kafka 主题.当消费者无法使用 Kafka 时，我看到了很多关于应用程序启动的信息，但关于生产者的信息并不多.我的应用将同时执行这两项操作.

edit: it is a temporary solution until we are able to deploy Kafka in this environment. The app does not produce and consume its own records. It is one part of a multi app deployment where each app both produces for other apps and consumes other apps Kafka topics. I see a lot of info around app startup when Kafka is not available for a consumer, but not much is out there with regards to producers. My app will be doing both.

推荐答案

这种应用程序的目的是什么(生成和使用自己的记录)?嵌入式代理不是为生产使用而设计的.

What would be the purpose of such an application (produce and consume its own records)? The embedded broker is not designed for production use.

从 2.3.4 版本开始，容器属性 missingTopicsFatal 默认为 false，即使代理不可用，也允许容器启动.对于早期版本，您可以将其设置为 false 以获得相同的效果.

Since version 2.3.4, the container property missingTopicsFatal is false by default, which will allow the container to start even if the broker is not available. With earlier versions, you can set it to false to get the same effect.

当它为 true 时，容器在启动期间连接到代理以验证主题是否存在.

When it is true, the container connects to the broker during startup to verify that the topic(s) exist.

您还可以设置容器的 autoStartup=false 以完全阻止容器启动.

You can also set the container's autoStartup=false to prevent the container from starting at all.

编辑

我不建议在生产中使用它，但是您可以从 spring-kafka-test 中删除 test 范围并将代理声明为 @豆...

I wouldn't recommend using this in production, but you can remove the test scope from spring-kafka-test and declare the broker as a @Bean...

        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka-test</artifactId>
<!--            <scope>test</scope> -->
        </dependency>

@Bean
EmbeddedKafkaBroker broker() {
    return new EmbeddedKafkaBroker(1)
            .kafkaPorts(9092)
            .brokerListProperty("spring.kafka.bootstrap-servers"); // override application property
}

我刚刚用这个应用程序测试过...

I just tested it with this app...

@SpringBootApplication
public class So63812994Application {

    public static void main(String[] args) {
        SpringApplication.run(So63812994Application.class, args);
    }

    @Bean
    EmbeddedKafkaBroker broker() {
        return new EmbeddedKafkaBroker(1)
                .kafkaPorts(9092)
                .brokerListProperty("spring.kafka.bootstrap-servers");
    }

    @Bean
    public NewTopic topic() {
        return TopicBuilder.name("so63812994").partitions(1).replicas(1).build();
    }

    @KafkaListener(id = "so63812994", topics = "so63812994")
    public void listen(String in) {
        System.out.println(in);
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate<String, String> template) {
        return args -> {
            template.send("so63812994", "foo");
        };
    }

}

spring.kafka.bootstrap-servers=realKafka:9092
spring.kafka.consumer.auto-offset-reset=earliest

EDIT2

通过上述配置，同一主机上的其他应用程序可以通过localhost:9092进行连接.

With the above configuration, other applications on the same host can connect with localhost:9092.

如果您需要远程访问这个嵌入式代理，您将需要一些额外的配置:

If you need remote access to this embedded broker, you will need some additional configuration:

@Bean
EmbeddedKafkaBroker broker() {
    return new EmbeddedKafkaBroker(1)
            .kafkaPorts(9092)
            .brokerProperty("listeners", "PLAINTEXT://localhost:9092,REMOTE://10.0.0.20:9093")
            .brokerProperty("advertised.listeners", "PLAINTEXT://localhost:9092,REMOTE://10.0.0.20:9093")
            .brokerProperty("listener.security.protocol.map", "PLAINTEXT:PLAINTEXT,REMOTE:PLAINTEXT")
            .brokerListProperty("spring.kafka.bootstrap-servers");
}

然后您可以使用 10.0.0.20:9093 从其他服务器连接.

You can then connect from other servers with 10.0.0.20:9093.

这篇关于如何在内存或嵌入式 kafka 中实现而不用于测试目的?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！