杀死风暴拓扑后的资源清理 [英] Resource clean up after killing storm topology

问题描述

我们有一个与 MariaDB 数据库交互的风暴拓扑.我们的 Bolt 实现了 IRichBolt 接口并覆盖生命周期方法.我们在 prepare 方法中打开一个数据库连接，并在 cleanup 方法中关闭它.cleanup 方法文档说:

We have a storm topology which interacts with a MariaDB database. Our Bolts implement the IRichBolt interface and override the lifecycle methods. We open a db connection in our prepare method and close it in the cleanup method. The cleanup method documentation says:

当 IBolt 将要关闭时调用.不保证会调用清理，因为主管在集群上 kill -9 的工作进程.保证调用清理的一个上下文是在本地模式下运行 Storm 时拓扑被终止

Called when an IBolt is going to be shutdown. There is no guarentee that cleanup will be called, because the supervisor kill -9's worker processes on the cluster. The one context where cleanup is guaranteed to be called is when a topology is killed when running Storm in local mode

并且 kill -9 命令会在不清理任何资源的情况下终止进程.所以我们得出这个结论，在杀死拓扑时，没有必要调用 cleanup 方法并且数据库连接将被关闭.

And the kill -9 command kills the process without cleaning up any resources. So we have come to this conclusion that on killing the topology it is not necessary that the cleanup method would be called and the db connection will be closed.

所以继续我的问题，我们有一个用于拓扑部署的 shell 脚本，它在执行时会以 0 超时终止当前拓扑并部署一个新拓扑.我们在 db 级别面临一个问题，即打开了许多连接，这给我们暗示以前的连接没有关闭.(上一个拓扑中打开的那个).

So moving forward to my question, we have a shell script for topology deployment which when executed kills the current topology with a timeout of 0 and deploys a new topology. We are facing an issue at db level that there are many connections opened which gives us the hint that previous connections were not closed. (The one opened in the previous topology).

我们的假设正确吗?增加超时会清除所有资源吗?

Is our assumption correct? Will increasing the timeout clean up all the resources?

推荐答案

没有.增加拓扑超时不会影响您的工作人员必须清理的时间.当您使用例如一个 30 秒的超时时间，它只是关闭了 spout，并让拓扑的其余部分有 30 秒的时间来完成处理.

No. Increasing the topology timeout won't affect how long your worker has to clean up. When you use e.g. a 30s timeout, that just shuts off the spout and gives the rest of the topology 30 secs to finish processing.

您想要的是增加 Storm 允许工人在强制杀死之前关闭的时间.https://github.com/apache/storm/blob/b07413670fa62fec077c92cb78fc711c3bda820c/storm-server/src/main/java/org/apache/storm/DaemonConfig.java#L780 选项允许你指定风暴持续多长时间将在发送 kill -9 之前等待.默认值为 3 秒.需要在storm.yaml中设置该选项，在拓扑配置中设置不会有任何影响.

What you want is to increase how long Storm allows a worker to spend shutting down before force killing. The https://github.com/apache/storm/blob/b07413670fa62fec077c92cb78fc711c3bda820c/storm-server/src/main/java/org/apache/storm/DaemonConfig.java#L780 option allows you to specify how long Storm will wait before sending the kill -9. The default is 3 seconds. You need to set this option in storm.yaml, setting it in the topology configuration will have no effect.

这篇关于杀死风暴拓扑后的资源清理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！