关于 flink 任务管理器和任务槽的核心分配 [英] About core allocation for flink task manager and task slot

问题描述

我使用以下命令启动 flink yarn 会话.

I use following command to kick off a flink yarn session.

yarn-session.sh -jm 4096 -tm 4096 -n 4 -s 2

上面的命令意味着将启动4个任务管理器(也意味着启动了4个纱线容器，因为每个任务管理器都是一个纱线容器)，每个任务管理器有2个槽.

With above command, it means that 4 task managers will be started(also means that 4 yarn containers are started since every task manager is a yarn container), and 2 slots for each task manager.

由于一个任务管理器是一个纱线容器，也就是说，每个任务管理器只分配一个核心，但我为每个任务管理器指定了2个插槽，这意味着这两个插槽将只共享一个核心?

Since one task manager is a yarn container, that means, only one core is allocated for each task manager , but I have specified 2 slots for each task manager, which means the two slots will share only one core?

我想过每个槽至少要占用一个核心吗?

I have thought that each slot should occupy at least one core?

不确定我是否理解正确或我错过了什么.

Not sure I have understood correctly or I have missed something.

推荐答案

每个插槽一个内核"已被建议作为默认起点，但不应将其视为规则.你可以自由地做任何你想做的事.如果每个插槽包含多个计算密集型任务，则为每个插槽提供多个内核是有意义的.另一方面，如果每个插槽几乎没有工作要做，并且将大部分时间都花在 I/O 上，那么小核就足够了.

"One core per slot" has been suggested as a default starting point, but it shouldn't be considered a rule. You are free to do whatever you want. If each slot contains several computationally intensive tasks, it makes sense to give each slot multiple cores. On the other hand, if each slot has little work to do, and will spend most of its time blocked on i/o, a fractional core will be enough.

这篇关于关于 flink 任务管理器和任务槽的核心分配的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！