在ECS Fargate中操作芹菜工人 [英] Operating the Celery Worker in the ECS Fargate

问题描述

我正在使用AWS ECS进行项目.我想将Celery用作分布式任务队列.可以将Celery Worker设置为EC2类型，但是由于实例处于空闲状态的时间较长，因此我认为AWS Fargate运行作业并立即退出将具有成本效益.

I am working on a project using AWS ECS. I want to use Celery as a distributed task queue. Celery Worker can be build up as EC2 type, but because of the large amount of time that the instance is in the idle state, I think it would be cost-effective for AWS Fargate to run the job and quit immediately.

您是否对如何在AWS云中有效使用Celery Worker提出建议?

Do you have suggestions on how to use the Celery Worker efficiently in the AWS cloud?

推荐答案

Fargate启动类型的启动时间要比EC2启动类型的启动时间长，因为启动任务时AWS会为您完成所有主机操作" ，包括臭名昭著的ENI连接缓慢，以及可能从Docker存储库下载映像.现在没有任何竞争，每次EC2启动类型都会更快.

Fargate launch type is going to take longer to spin up than EC2 launch type, because AWS is doing all the "host things" for you when you start the task, including the notoriously slow attaching of an ENI, and likely downloading the image from a Docker repo. Right now there's no contest, EC2 launch type is faster every time.

因此，这实际上取决于您希望工人从事的工作类型.由于上述原因，您可以期望新的Fargate任务花费几分钟进入RUNNING状态.另一方面，由于ENI已在主机上安装并且映像已下载(最好)或已下载最多(可能是最差)，因此EC2启动将很快从PENDING变为RUNNING.

So it really depends on the type of work you want the workers to do. You can expect a new Fargate task to take a few minutes to enter a RUNNING state for the aforementioned reasons. EC2 launch, on the other hand, because the ENI is already in place on your host and the image is already downloaded (at best) or mostly downloaded (likely worst), will move from PENDING to RUNNING very quickly.

这是当前流行的智慧，通常被讨论为成本因素，因为Fargate无法利用预留实例和现货定价等典型的EC2成本节省机制.与EC2相比，一直运行Fargate的成本很高.

This is the current prevailing wisdom, often discussed as a cost factor because Fargate can't take advantage of the typical EC2 cost savings mechanisms like reserved instances and spot pricing. It's expensive to run Fargate all the time, compared to EC2.

很明显，在Fargate中运行100％完全可以(我们愿意)，但是您必须愿意接受这样做的缺点-扩展速度和成本降低.

To be clear, it's perfectly fine to run 100% in Fargate (we do), but you have to be willing to accept the downsides of doing that - slower scaling and cost.

注意，您可以在同一集群中运行两种启动类型.无论如何，集群都是合乎逻辑的，只是一种组织资源的方式.

Note you can run both launch types in the same cluster. Clusters are logical anyway, just a way to organize your resources.

此示例显示了一个静态EC2启动类型服务，该服务运行4个celery任务.任务的数量，规格，实例大小以及所有这些都无关紧要，但是您可以随心所欲地完成.重要的是-EC2启动类型服务无需扩展； Fargate启动类型服务能够根据您的扩​​展规则，从无运行(在几乎没有工作要做或没有工作的时期)扩展到您可以处理的尽可能多的工作人员.

This example shows a static EC2 launch type service running 4 celery tasks. The number of tasks, specs, instance size and all doesn't really matter, do it up however you like. The important thing is - EC2 launch type service doesn't need to scale; the Fargate launch type service is able to scale from nothing running (during periods where there's little or no work to do) to as many workers as you can handle, based on your scaling rules.

运行1个EC2启动类型t3.medium(2vcpu/4GB).

Running 1 EC2 launch type t3.medium (2vcpu/4GB).

最小任务:2，所需任务:4，最大任务:4

Min tasks: 2, Desired: 4, Max tasks: 4

在此EC2启动类型中，以512/1024运行4个芹菜任务.

Running 4 celery tasks at 512/1024 in this EC2 launch type.

无扩展政策

最小任务:0，所需:(x)，最大任务:32

Min tasks: 0, Desired: (x), Max tasks: 32

以512/1024的速度运行(x)个芹菜任务(与EC2启动类型相同的任务定义)

Running (x) celery tasks (same task def as EC2 launch type) at 512/1024

向此服务添加扩展策略

这篇关于在ECS Fargate中操作芹菜工人的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！