AWS Glue中1个DPU中的最大并发任务数 [英] Maximum number of concurrent tasks in 1 DPU in AWS Glue

问题描述

AWS Glue中的标准DPU带有4个vCPU和2个执行程序. 我对可以与此配置并行运行的并发任务的最大数量感到困惑.在具有4vcpu和2个执行程序的单个DPU上是4还是8?

A standard DPU in AWS Glue comes with 4 vCPU and 2 executors. I am confused about the maximum number of concurrent tasks that can be run in parallel with this configuration. Is it 4 or 8 on a single DPU with 4vcpu and 2 executors?

推荐答案

我与AWS Glue支持团队进行了类似的讨论，我将与您分享他们对Glue Configuration的了解.以Standard和G1.X配置为例.

I had a similar discussion with the AWS Glue support team about this, I'll share with you what they told me about Glue Configuration. Take in example the Standard and the G1.X configuration.

标准DPU配置:

• 为MasterNode保留1个DPU
• 为驱动程序/ApplicationMaster保留1个执行器
• 每个DPU配置有2个执行器
• 每个执行程序都配置有5.5 GB内存
• 每个执行器都配置有4个内核

G.1X WorkerType配置:

G.1X WorkerType Configuration:

• 为MasterNode添加了1个DPU
• 为驱动程序/ApplicationMaster保留1个DPU
• 每个工作程序都配置了1个执行程序
• 每个执行程序都配置有10 GB内存
• 每个执行器都配置有8个内核

例如，如果我们有一个具有21 DPU标准配置的作业，则意味着我们拥有:

If we have for example a Job with Standard Configuration with 21 DPU means that we have:

• 为Master保留1个DPU
• 20个DPU x 2 = 40个执行程序
• 1个驱动程序/AM-40个执行程序= 39个执行程序

然后，我们最终得到了156个核心.意思是，您的工作有156个执行插槽.例如，如果您从S3中读取文件，则意味着您将能够并行接受156个输入文件.

Which we then end up with a total amount of 156 cores. Meaning, your job has 156 slots for execution. If for example you read files from S3 that means that you will be able to accept 156 input files in parallel.

希望有帮助.

这篇关于AWS Glue中1个DPU中的最大并发任务数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！