关于flink任务管理器和任务槽的核心分配 [英] About core allocation for flink task manager and task slot

问题描述

我使用以下命令来启动flink纱线会话.

I use following command to kick off a flink yarn session.

yarn-session.sh -jm 4096 -tm 4096 -n 4 -s 2

使用上述命令，这意味着将启动4个任务管理器(由于每个任务管理器都是一个纱线容器，因此也将启动4个纱线容器)，并为每个任务管理器分配2个插槽.

With above command, it means that 4 task managers will be started(also means that 4 yarn containers are started since every task manager is a yarn container), and 2 slots for each task manager.

由于一个任务管理器是一个纱线容器，这意味着每个任务管理器仅分配了一个核心，但是我为每个任务管理器指定了2个插槽，这意味着这两个插槽将仅共享一个核心?

Since one task manager is a yarn container, that means, only one core is allocated for each task manager , but I have specified 2 slots for each task manager, which means the two slots will share only one core?

我以为每个插槽应该至少占据一个核心?

I have thought that each slot should occupy at least one core?

不确定我是否正确理解或错过了某些东西.

Not sure I have understood correctly or I have missed something.

推荐答案

每个插槽一个核心"建议将其作为默认起点，但不应将其视为规则.您可以自由地做任何您想做的事.如果每个插槽包含多个计算密集型任务，则为每个插槽分配多个内核是有意义的.另一方面，如果每个插槽的工作量很少，并且将大部分时间都用在I/O上，那么分数内核就足够了.

"One core per slot" has been suggested as a default starting point, but it shouldn't be considered a rule. You are free to do whatever you want. If each slot contains several computationally intensive tasks, it makes sense to give each slot multiple cores. On the other hand, if each slot has little work to do, and will spend most of its time blocked on i/o, a fractional core will be enough.

这篇关于关于flink任务管理器和任务槽的核心分配的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！