是否有可能在Apache Storm中在运行时动态添加任务，而不仅仅是重新平衡执行者 [英] is it possible to add Tasks dynamically at runtime in apache storm not just rebalance executors

问题描述

我需要一个据我所知(基于文档)尚未实现的功能.我需要在运行时添加更多任务，而不需要最初的大量任务，因为这可能会导致性能问题.因为每个执行者执行一个以上的任务不会提高并行度-执行者总是有一个线程用于其所有任务，这意味着任务在执行者上串行运行.

I need a functionality in storm that i know (based on the docs) has not been yet implemented. I need to add more tasks at runtime without the need to have an initial large number of tasks, because it might cause performance issues. because Running more than one task per executor does not increase the level of parallelism -- an executor always has one thread that it uses for all of its tasks, which means that tasks run serially on an executor.

我知道rebalance命令可用于在运行时添加执行程序和工作进程，并且有一个规则，即#executors< =#tasks，这意味着任务数量在运行时应该是静态的，但是我很好奇很难(如果不是不可能)将此功能添加到风暴中.

I know that rebalance command can be used to add executors ans worker processes at runtime and there is a rule that #executors<=#tasks and this means that number of tasks should be static at runtime, but i'm curious how hard is it(if not impossible) to add this feature to storm.

是否有办法在风暴中实现此功能，或者根本无法实现?如果有办法，请给我提示如何做.

Is there a way to implement this functionality in storm or it can't be done at all? if there is a way please give me clue how to do it.

推荐答案

由于这些额外的任务是串行运行的，所以不确定您的意思".

Not sure what you mean by "since those extra tasks run serially".

Tasks is Storm用于利用数据并行性.从理论上讲，可以添加代码以在运行时更改任务数.但这将是一个巨大的变化，并且AFAIK没有计划添加此功能.

Tasks is Storm are use to exploit data parallelism. In theory it's possible to add code to change the number of tasks at runtime. But it would be a huge change and AFAIK there are no plans to add this feature.

比较 http://storm.apache.org/releases/1.0.3/Understanding-the-parallelism-of-a-Storm-topology.html

由于将密钥分配给基于哈希的任务，因此更改任务数将需要将所有密钥重新哈希为新任务.如果操作员建立了基于密钥的内部状态，则也需要按密钥对这种状态进行分区并相应地重新分配.

Because keys are assigned to tasks hash based, changing the number of tasks would require to rehash all keys to new tasks. If an operator builds up an key-based internal state, this state would need to get partitioned by key and redistributed accordingly, too.

这篇关于是否有可能在Apache Storm中在运行时动态添加任务，而不仅仅是重新平衡执行者的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！