AWS批处理始终为每个作业启动新的ec2实例 [英] AWS batch to always launch new ec2 instance for each job
问题描述
我已使用
- 托管计算环境
- 工作队列
- 职位定义
实际的工作(docker容器)进行了大量的视频编码，因此占用了大部分CPU.该过程本身需要几分钟(将近5分钟才能初始化所有编码器).理想情况下，我希望每个实例完成一项工作，以使编码器不会出现CPU短缺的情况.
The actual job(docker container) does a lot of video encoding and hence uses up most of the CPU. The process itself takes a few minutes (close to 5 minutes to get all the encoders initialized). Ideally I would want one job per instance so that the encoders are not CPU starved.
我的问题是，当我同时启动多个作业或关闭足够多的作业时，AWS批处理决定在同一实例中启动两个作业，因为第一个容器仍在初始化且尚未开始使用CPU. 在我看来，两个工作都将创建的实例视为可用状态是一种竞争条件.
My issue is when I launch multiple jobs at the same time or close enough, AWS batch decides launch both of them in the same instance as the first container is still initializing and has not started using CPUs yet. It seems like a race condition to me where both jobs see the instance created as available.
是否有一种方法可以为每个作业启动一个实例，而无需查找已经在运行的实例?或任何其他解决方案可以在实例被指定用于特定作业后将其锁定?
Is there a way I can launch one instance for each job without looking for instances that are already running? Or any other solution to lock an instance once it is designated for a particular job?
非常感谢您的帮助.
推荐答案
您不必担心将作业分离到不同的实例上，因为作业所运行的容器受其可以使用的vCPU数量的限制.例如，如果启动两个分别需要4个vCPU的作业，则Batch可能启动具有8个vCPU的实例，并在同一实例上运行这两个作业.每个作业只能访问4个vCPU，因此性能应与实例上没有其他作业的单独运行的作业相同.
You shouldn't have to worry about separating the jobs onto different instances because the containers the jobs run in are limited in how many vCPUs they can use. For example, if you launch two jobs that each require 4 vCPUs, Batch might spin up an instance that has 8 vCPUs and run both jobs on the same instance. Each job will have access to only 4 of the vCPUs, so performance should be identical to a job running on its own with no other jobs on the instance.
但是，如果仍然要将作业分离到单独的实例中，则可以通过将作业的vCPU与计算环境中的实例类型进行匹配来实现.例如，如果您有一个需要4个vCPU的作业，则可以将计算环境配置为仅允许c5.xlarge实例，因此每个实例只能运行一个作业.但是，如果要运行具有更高vCPU要求的其他作业，则必须在不同的计算环境中运行它们.
However, if you still want to separate the jobs onto separate instances, you can do so by matching the vCPUs of the job with the instance type in the compute environment. For example, if you have a job that requires 4 vCPUs, you can configure your compute environment to only allow c5.xlarge instances, so each instance can run only one job. However, if you want to run other jobs with higher vCPU requirements, you would have to run them in a different compute environment.
这篇关于AWS批处理始终为每个作业启动新的ec2实例的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
