我如何在内存或嵌入式kafka中实现非测试目的? [英] How do I implement in memory or embedded kafka not for testing purposes?

问题描述

我希望在仅允许Kafka由上述应用程序运行的环境中部署Spring Boot应用程序.我的应用将是Kafka的生产者和消费者.有没有一种方法可以在启动时运行内存中实例，该实例可用于测试以外的目的?另外，是否有一种方法可以启动Spring Boot应用程序，如果它不能以生产者和消费者身份连接到Kafka，也不会失败?

I am looking to deploy a spring boot app in an environment that only allows Kafka to be run by the aforementioned app. My app will be a Kafka producer and consumer. Is there a way to run an in memory instance on startup that can be used for purposes other than testing? Alternatively, is there a way to startup a spring boot app that will not fail if it cannot connect to Kafka as a producer and consumer?

edit:在我们能够在此环境中部署Kafka之前，这是一个临时解决方案.该应用程序不会产生和使用自己的记录.它是多应用程序部署的一部分，其中每个应用程序既为其他应用程序生成又使用其他应用程序Kafka主题.当消费者无法使用Kafka时，我会看到很多有关应用程序启动的信息，但对于生产者而言却没有太多信息.我的应用将同时执行这两项操作.

edit: it is a temporary solution until we are able to deploy Kafka in this environment. The app does not produce and consume its own records. It is one part of a multi app deployment where each app both produces for other apps and consumes other apps Kafka topics. I see a lot of info around app startup when Kafka is not available for a consumer, but not much is out there with regards to producers. My app will be doing both.

推荐答案

这样的应用程序(产生和使用自己的记录)的目的是什么?嵌入式代理并非设计用于生产用途.

What would be the purpose of such an application (produce and consume its own records)? The embedded broker is not designed for production use.

从2.3.4版本开始，默认情况下容器属性 missingTopicsFatal 为false，即使代理不可用，该属性也将允许容器启动.在早期版本中，您可以将其设置为false以获得相同的效果.

Since version 2.3.4, the container property missingTopicsFatal is false by default, which will allow the container to start even if the broker is not available. With earlier versions, you can set it to false to get the same effect.

为true时，容器在启动期间连接到代理，以验证主题是否存在.

When it is true, the container connects to the broker during startup to verify that the topic(s) exist.

您还可以设置容器的 autoStartup = false 来完全阻止容器启动.

You can also set the container's autoStartup=false to prevent the container from starting at all.

编辑

我不建议在生产环境中使用它，但是您可以从 spring-kafka-test 中删除 test 范围，并将代理声明为 @Bean ...

I wouldn't recommend using this in production, but you can remove the test scope from spring-kafka-test and declare the broker as a @Bean...

        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka-test</artifactId>
<!--            <scope>test</scope> -->
        </dependency>

@Bean
EmbeddedKafkaBroker broker() {
    return new EmbeddedKafkaBroker(1)
            .kafkaPorts(9092)
            .brokerListProperty("spring.kafka.bootstrap-servers"); // override application property
}

我刚刚使用此应用程序对其进行了测试...

I just tested it with this app...

@SpringBootApplication
public class So63812994Application {

    public static void main(String[] args) {
        SpringApplication.run(So63812994Application.class, args);
    }

    @Bean
    EmbeddedKafkaBroker broker() {
        return new EmbeddedKafkaBroker(1)
                .kafkaPorts(9092)
                .brokerListProperty("spring.kafka.bootstrap-servers");
    }

    @Bean
    public NewTopic topic() {
        return TopicBuilder.name("so63812994").partitions(1).replicas(1).build();
    }

    @KafkaListener(id = "so63812994", topics = "so63812994")
    public void listen(String in) {
        System.out.println(in);
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate<String, String> template) {
        return args -> {
            template.send("so63812994", "foo");
        };
    }

}

spring.kafka.bootstrap-servers=realKafka:9092
spring.kafka.consumer.auto-offset-reset=earliest

EDIT2

使用上述配置，同一主机上的其他应用程序可以与 localhost:9092 连接.

With the above configuration, other applications on the same host can connect with localhost:9092.

如果您需要对该嵌入式代理的远程访问，则需要一些其他配置:

If you need remote access to this embedded broker, you will need some additional configuration:

@Bean
EmbeddedKafkaBroker broker() {
    return new EmbeddedKafkaBroker(1)
            .kafkaPorts(9092)
            .brokerProperty("listeners", "PLAINTEXT://localhost:9092,REMOTE://10.0.0.20:9093")
            .brokerProperty("advertised.listeners", "PLAINTEXT://localhost:9092,REMOTE://10.0.0.20:9093")
            .brokerProperty("listener.security.protocol.map", "PLAINTEXT:PLAINTEXT,REMOTE:PLAINTEXT")
            .brokerListProperty("spring.kafka.bootstrap-servers");
}

然后您可以使用 10.0.0.20:9093 从其他服务器进行连接.

You can then connect from other servers with 10.0.0.20:9093.

这篇关于我如何在内存或嵌入式kafka中实现非测试目的?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！