如何通过以编程方式插入消息来 E2E 测试 Storm Topology 的功能 [英] How to E2E test functionality of Storm Topology by programmatically inserting messages

问题描述

我们的 Apache Storm 拓扑使用 KafkaSpout 侦听来自 Kafka 的消息，并在做了大量映射/减少/丰富/聚合等之后最终将数据插入到 Cassandra 中.还有另一个 kafka 输入，如果拓扑找到响应，我们将接收用户对数据的查询，然后将其发送到第三个 kafka 主题.现在我们想使用 Junit 编写 E2E 测试，其中我们可以直接以编程方式将数据插入到拓扑中，然后通过插入用户查询消息，我们可以在第三点断言我们的查询收到的响应是正确的.

Our Apache Storm topology listens messages from Kafka using KafkaSpout and after doing lot of mapping/reducing/enrichment/aggregation etc. etc finally inserts data into Cassandra. There is another kafka input where we receive user queries for data if topology finds a response then it sends that onto a third kafka topic. Now we want to write E2E test using Junit in which we can directly programmatically insert data into topology and then by inserting user query message, we can assert on third point that response received on our query is correct.

为了实现这一点，我们考虑启动 EmbeddedKafka 和 CassandraUnit，然后用它们替换实际的 Kafka 和 Cassandra，然后我们可以在这个单一的 Junit 测试的上下文中启动拓扑.

To achieve this, we thought of starting EmbeddedKafka and CassandraUnit and then replacing actual Kafka and Cassandra with them and then we can start topology in the context of this single Junit test.

但是我们的方法并不适合 JUnit，因为它使这些测试变得过于庞大.启动 kafka、cassandra 和拓扑都需要时间并消耗大量资源.Apache Storm 中是否有任何东西可以支持我们计划编写的测试类型?

But our approach doesn't fit well with JUnit because it makes these tests too bulky. Starting kafka, cassandra and topology all are time taking and consume lot of resource. Is there anything in Apache Storm which can support kind of testing we are planning to write?

推荐答案

这里有多种选择，具体取决于您可以忍受的减速类型:

There are a number of options you have here, depending on what kind of slowdown you can live with:

• 正如您所提到的，您可以启动 Kafka、Cassandra 和拓扑.这是最慢的选择，也是最现实的".

• As you mentioned, you can start Kafka, Cassandra and the topology. This is the slowest option, and the "most realistic".

启动 Kafka 和 Cassandra 一次，并在所有测试中重复使用它们.您可以对 Storm LocalCluster 执行相同的操作.在每次测试之间清除 Kafka/Cas​​sandra 可能会更快(例如删除所有主题)，而不是重新启动它们.

Start Kafka and Cassandra once, and reuse them for all the tests. You can do the same with the Storm LocalCluster. It is likely faster to clear Kafka/Cassandra between each test (e.g. deleting all topics) instead of restarting them.

在测试中用存根替换 Kafka spouts/bolts 和 Cassandra bolt.Storm 内置了许多用于固定螺栓和喷口的工具，例如FixedTupleSpout、FeederSpout、LocalCluster 中的跟踪拓扑和可完成拓扑功能.通过这种方式，您可以将一些固定的元组插入到拓扑中，并对发送到 Cassandra bolt 存根的元组进行断言.有一些此功能的示例 此处 和这里

Replace the Kafka spouts/bolts and Cassandra bolt with stubs in test. Storm has a number of tools built in for stubbing bolts and spouts, e.g. the FixedTupleSpout, FeederSpout, the tracked topology and completable topology functionality in LocalCluster. This way you can insert some fixed tuples into the topology, and do asserts about which tuples where sent to the Cassandra bolt stub. There's examples of some of this functionality here and here

最后，您当然可以对单个螺栓进行单元测试.这是最快的一种测试.您可以使用 Testing.testTuple 创建测试元组以传递给螺栓.

Finally you can of course unit test individual bolts. This is the fastest kind of test. You can use Testing.testTuple to create test tuples to pass to the bolt.

这篇关于如何通过以编程方式插入消息来 E2E 测试 Storm Topology 的功能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！