Spark 执行器 &任务并发 [英] Spark executor & tasks concurrency

问题描述

在 Spark 中，一个 executor 可能同时运行许多任务，可能是 2 个、5 个或 6 个.

In Spark, an executor may run many tasks concurrently maybe 2 or 5 or 6 .

Spark 如何计算(或计算)在同一个 executor 中同时运行的任务数量，即一个 executor 中可以同时运行多少个任务?

How Spark figures out (or calculate) the number of tasks to be run in the same executor concurrently i.e how many tasks can run in an executor concurrently?

一个执行器可能正在执行一项任务，但可能会在同一个执行器上同时运行另一个任务?这样做的标准是什么?

An executor may be executing one task but one more task maybe be placed to run concurrently on same executor? What's the criteria for that?

一个执行器有固定数量的内核 &记忆.由于我们没有指定内存 &Spark 中任务的核心数要求，如何计算一个执行器中可以同时运行的核心数?

An executor has fixed number of cores & memory. As we do not specify memory & cores requirements for task in Spark, how to calculate how many can run concurrently in an executor?

推荐答案

在一个 executor 中并行运行的任务数 = 配置的内核数.您可以随时通过配置更改此编号.执行器整体(并行或顺序)运行的任务总数取决于创建的任务总数(通过拆分的数量)和通过执行器的数量.

The number of tasks run parallely within an executor = number of cores configured. You can always change this number through configuration. The total number of tasks run by executor overall ( parallel or sequential) depends upon the total number of tasks created ( through number of splits) and through number of executors.

在一个执行器中运行的所有任务共享配置的相同内存.在内部，它只是启动与内核数量一样多的线程.

All tasks running in one executor share the same memory configured. Inside, it just launches as many threads as number of cores.

这篇关于Spark 执行器 &任务并发的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！