从 spout 中杀死风暴拓扑 [英] Killing storm topology from spout

问题描述

我们有一个用例，我们不想连续运行风暴拓扑.相反，有一组输入(10K+)应该在指定的时间处理，Spout 不断地发出这些输入并由拓扑中的其余螺栓处理.处理完所有输入后，我的 spout 中的 nextTuple 将没有任何内容可发出.

We have an use case where we do not want to run storm topology continuously. Instead, there are set of inputs( 10K+) that should be processed at the specified time, Spout continuously emits these inputs and get processed by rest of the bolts in the topology. Once all the inputs are processed, there is nothing to emit from nextTuple in my spout.

此时我们希望我们的拓扑在每天晚上 12:00 进入睡眠状态并重新启动进程.

At this time we wanted our topology to go to sleep and restart the process everyday night 12:00 am.

风暴配置中是否有任何属性可以设置为每天运行一次拓扑并在处理完成后休眠并在指定时间启动?

Is there any property to set in the storm config to run the topology once a day and sleep after processing is done and start at the specified time?

推荐答案

我不知道您要求的功能.Storm 不是批处理系统，它旨在连续运行.考虑 Storm 是否非常适合此用例.

I'm not aware of a feature like what you're asking for. Storm isn't a batch processing system, it's meant to be running continuously. Consider if Storm is a great fit for this use case.

也就是说，您应该能够实现您想要的.您可以在 spout 输入的末尾输入我已完成"消息.当 spout 命中该消息并且所有其他挂起的消息都被确认时，它可以使用 Nimbus 客户端来终止或停用拓扑(取决于您是要终止还是停用)，请参阅 https://stackoverflow.com/a/37134473/8845188.然后最后一步是使用您最喜欢的调度软件在每天午夜重新提交/重新激活拓扑.

That said, you should be able to implement what you want. You could put in an "I'm done" message at the end of your spout input. When the spout hits that message and all other pending messages are acked, it could use the Nimbus client to kill or deactivate the topology (depending on whether you want to kill or deactivate), see https://stackoverflow.com/a/37134473/8845188. Then the final step would be using your favorite scheduling software to resubmit/reactivate the topology every day at midnight.

这篇关于从 spout 中杀死风暴拓扑的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！