风暴最大喷口待定 [英] Storm max spout pending

查看:27
本文介绍了风暴最大喷口待定的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

这是一个关于 Storm 的 max spout pending 工作原理的问题.我目前有一个 spout 读取文件并为文件中的每一行发出一个元组(我知道 Storm 不是处理文件的最佳解决方案,但我没有选择解决这个问题).

This is a question regarding how Storm's max spout pending works. I currently have a spout that reads a file and emits a tuple for each line in the file (I know Storm is not the best solution for dealing with files but I do not have a choice for this problem).

我将 topology.max.spout.pending 设置为 50k,以限制进入要处理的拓扑的元组数量.但是,我看到这个数字对拓扑没有任何影响.我看到每次都发出文件中的所有记录.我的猜测是这可能是由于我在 nextTuple() 方法中有一个循环,它发出文件中的所有记录.

I set the topology.max.spout.pending to 50k to throttle how many tuples get into the topology to be processed. However, I see this number not having any effect in the topology. I see all records in a file being emitted every time. My guess is this might be due to a loop I have in the nextTuple() method that emits all records in a file.

我的问题是:当达到 topology.max.spout.pending 时,Storm 是否会停止为 Spout 任务调用 nextTuple()?这是否意味着每次调用该方法时我应该只发出一个元组?

My question is: Does Storm just stop calling nextTuple() for the Spout task when topology.max.spout.pending is reached? Does this mean I should only emit one tuple every time the method is called?

推荐答案

正是如此!Storm 只能通过下一个命令限制您的 spout,因此如果您在收到第一个 next 时传输所有内容,Storm 无法限制您的 spout.

Exactly! Storm can only limit your spout with the next command, so if you transmit everything when you receive the first next, there is no way for Storm to throttle your spout.

Storm 开发人员建议使用单个 next 命令发出单个元组.然后,Storm 框架将根据需要限制您的 spout,以满足最大 spout 挂起"要求.如果您要发出大量元组,您最多可以将您的发出分批到最大 spout 的十分之一,让 Storm 有机会进行节流.

The Storm developers recommend emitting a single tuple with a single next command. The Storm framework will then throttle your spout as needed to meet the "max spout pending" requirement. If you're emitting a high number of tuples, you can batch your emits to at most a tenth of your max spout pending, to give Storm the chance to throttle.

这篇关于风暴最大喷口待定的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆