火花中的应用，工作，阶段和任务的概念是什么? [英] What is the concept of application, job, stage and task in spark?
问题描述
我的理解正确吗?
-
应用程序: 一火花提交.
Application: one spark submit.
工作: 一旦进行了懒惰的评估，就找到了工作.
job: once a lazy evaluation happens, there is a job.
阶段: 它与随机播放和转换类型有关. 我很难理解舞台的边界.
stage: It is related to the shuffle and the transformation type. It is hard for me to understand the boundary of the stage.
任务: 这是单元操作.每个任务一次转换.每次转换一项任务.
task: It is unit operation. One transformation per task. One task per transformation.
Help希望增进这种了解.
Help wanted to improve this understanding.
推荐答案
主要功能是应用程序.
在RDD上调用动作时，将创建一个作业".作业是提交给Spark的工作.
When you invoke an action on an RDD, a "job" is created. Jobs are work submitted to Spark.
基于混洗边界，将工作分为阶段". 这可以帮助您理解.
Jobs are divided into "stages" based on the shuffle boundary. This can help you understand.
每个阶段都根据RDD中的分区数进一步划分为任务.因此，任务是Spark的最小工作单元.
Each stage is further divided into tasks based on the number of partitions in the RDD. So tasks are the smallest units of work for Spark.
这篇关于火花中的应用，工作，阶段和任务的概念是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
