是否可以在 apache Storm 中在运行时动态添加任务，而不仅仅是重新平衡执行程序 [英] is it possible to add Tasks dynamically at runtime in apache storm not just rebalance executors

问题描述

我需要一个我知道(基于文档)尚未实现的 Storm 功能.我需要在运行时添加更多任务而不需要初始大量任务，因为这可能会导致性能问题.因为每个 executor 运行多个任务并不会提高并行度——一个 executor 总是有一个线程用于它的所有任务，这意味着任务在一个 executor 上串行运行.

I need a functionality in storm that i know (based on the docs) has not been yet implemented. I need to add more tasks at runtime without the need to have an initial large number of tasks, because it might cause performance issues. because Running more than one task per executor does not increase the level of parallelism -- an executor always has one thread that it uses for all of its tasks, which means that tasks run serially on an executor.

我知道 rebalance 命令可用于在运行时添加执行程序和工作进程，并且有一条规则是 #executors<=#tasks，这意味着任务数量在运行时应该是静态的，但我很好奇如何将此功能添加到 Storm 中很难(如果不是不可能的话).

I know that rebalance command can be used to add executors ans worker processes at runtime and there is a rule that #executors<=#tasks and this means that number of tasks should be static at runtime, but i'm curious how hard is it(if not impossible) to add this feature to storm.

有没有办法在storm中实现这个功能或者根本做不到?如果有办法，请告诉我该怎么做.

Is there a way to implement this functionality in storm or it can't be done at all? if there is a way please give me clue how to do it.

推荐答案

不确定因为这些额外的任务是串行运行的"是什么意思.

Not sure what you mean by "since those extra tasks run serially".

Tasks is Storm 用于利用数据并行性.理论上，可以添加代码来在运行时更改任务数量.但这将是一个巨大的变化，AFAIK 没有计划添加此功能.

Tasks is Storm are use to exploit data parallelism. In theory it's possible to add code to change the number of tasks at runtime. But it would be a huge change and AFAIK there are no plans to add this feature.

比较 http://storm.apache.org/releases/1.0.3/Understanding-the-parallelism-of-a-Storm-topology.html

因为键是基于哈希分配给任务的，更改任务数量需要将所有键重新哈希到新任务.如果算子建立了一个基于密钥的内部状态，这个状态也需要通过密钥进行分区并相应地重新分配.

Because keys are assigned to tasks hash based, changing the number of tasks would require to rehash all keys to new tasks. If an operator builds up an key-based internal state, this state would need to get partitioned by key and redistributed accordingly, too.

这篇关于是否可以在 apache Storm 中在运行时动态添加任务，而不仅仅是重新平衡执行程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！