如何通过以编程方式插入消息来端到端测试Storm拓扑功能 [英] How to E2E test functionality of Storm Topology by programmatically inserting messages

问题描述

我们的Apache Storm拓扑使用KafkaSpout侦听来自Kafka的消息，并在进行大量映射/缩减/富集/聚合等之后，最终将数据插入Cassandra.还有另一个kafka输入，如果拓扑找到响应，我们将在其中接收用户对数据的查询，然后将其发送到第三个kafka主题.现在，我们要使用Junit编写E2E测试，在该测试中，我们可以以编程方式直接将数据插入拓扑，然后通过插入用户查询消息，可以在第三点断言查询中收到的响应是正确的.

Our Apache Storm topology listens messages from Kafka using KafkaSpout and after doing lot of mapping/reducing/enrichment/aggregation etc. etc finally inserts data into Cassandra. There is another kafka input where we receive user queries for data if topology finds a response then it sends that onto a third kafka topic. Now we want to write E2E test using Junit in which we can directly programmatically insert data into topology and then by inserting user query message, we can assert on third point that response received on our query is correct.

为此，我们考虑了启动EmbeddedKafka和CassandraUnit，然后用它们替换实际的Kafka和Cassandra，然后可以在此Junit测试的上下文中启动拓扑.

To achieve this, we thought of starting EmbeddedKafka and CassandraUnit and then replacing actual Kafka and Cassandra with them and then we can start topology in the context of this single Junit test.

但是我们的方法不适用于JUnit，因为它会使这些测试变得过于庞大.启动kafka，cassandra和topology都是很费时间的，并且会消耗大量资源. Apache Storm中有什么可以支持我们计划编写的测试?

But our approach doesn't fit well with JUnit because it makes these tests too bulky. Starting kafka, cassandra and topology all are time taking and consume lot of resource. Is there anything in Apache Storm which can support kind of testing we are planning to write?

推荐答案

根据您可以承受的减慢速度，这里有很多选择:

There are a number of options you have here, depending on what kind of slowdown you can live with:

• 如前所述，您可以启动Kafka，Cassandra和拓扑.这是最慢的选项，也是最现实的"选项.

• As you mentioned, you can start Kafka, Cassandra and the topology. This is the slowest option, and the "most realistic".

一次启动Kafka和Cassandra，并将其重新用于所有测试.您可以使用Storm LocalCluster执行相同的操作.在每个测试之间(例如删除所有主题)清除Kafka/Cassandra可能比重新启动它们更快.

Start Kafka and Cassandra once, and reuse them for all the tests. You can do the same with the Storm LocalCluster. It is likely faster to clear Kafka/Cassandra between each test (e.g. deleting all topics) instead of restarting them.

在测试中用短管代替Kafka喷嘴/螺栓和Cassandra螺栓. Storm内置了许多工具，可用于锚固螺栓和喷口，例如LocalCluster中的FixedTupleSpout，FeederSpout，跟踪的拓扑和可完成的拓扑功能.这样，您可以在拓扑中插入一些固定的元组，并断言将哪些元组发送到Cassandra螺栓桩.有一些此功能的示例

Replace the Kafka spouts/bolts and Cassandra bolt with stubs in test. Storm has a number of tools built in for stubbing bolts and spouts, e.g. the FixedTupleSpout, FeederSpout, the tracked topology and completable topology functionality in LocalCluster. This way you can insert some fixed tuples into the topology, and do asserts about which tuples where sent to the Cassandra bolt stub. There's examples of some of this functionality here and here

最后，您当然可以对单个螺栓进行单元测试.这是最快的一种测试.您可以使用Testing.testTuple创建测试元组以传递到螺栓.

Finally you can of course unit test individual bolts. This is the fastest kind of test. You can use Testing.testTuple to create test tuples to pass to the bolt.

这篇关于如何通过以编程方式插入消息来端到端测试Storm拓扑功能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！