您如何使用 AWS Elastic Beanstalk 运行工作线程? [英] How do you run a worker with AWS Elastic Beanstalk?

问题描述

我正在 AWS Elastic Beanstalk 上启动一个 Django 应用程序.我想运行后台任务或工作线程以运行 celery.

I am launching a Django application on AWS Elastic Beanstalk. I'd like to run a background task or worker in order to run celery.

我无法找到是否可能.如果是，如何实现?

I can not find if it is possible or not. If yes how could it be achieved?

这是我现在正在做的事情，但这每次都会产生一个事件类型错误.

Here is what I am doing right now, but this is producing an event type error every time.

container_commands:
  01_syncdb:
    command: "django-admin.py syncdb --noinput"
    leader_only: true
  50_sqs_email:
    command: "./manage.py celery worker --loglevel=info"
    leader_only: true

推荐答案

正如@chris-wheadon 在他的评论中所建议的，你应该尝试在后台运行 celery 作为守护进程.AWS Elastic Beanstalk 已经使用 supervisord 来运行一些守护进程.因此，您可以利用它来运行 celeryd 并避免为此创建自定义 AMI.对我来说效果很好.

As @chris-wheadon suggested in his comment, you should try to run celery as a deamon in the background. AWS Elastic Beanstalk uses supervisord already to run some deamon processes. So you can leverage that to run celeryd and avoid creating a custom AMI for this. It works nicely for me.

我所做的是在 EB 将应用程序部署到实例后，以编程方式将 celeryd 配置文件添加到实例中.棘手的部分是该文件需要为守护进程设置所需的环境变量(例如 AWS 访问密钥，如果您在应用中使用 S3 或其他服务).

What I do is to programatically add a celeryd config file to the instance after the app is deployed to it by EB. The tricky part is that the file needs to set the required environmental variables for the deamon (such as AWS access keys if you use S3 or other services in your app).

下面是我使用的脚本副本，将此脚本添加到配置 EB 环境的 .ebextensions 文件夹中.

Below there is a copy of the script that I use, add this script to your .ebextensions folder that configures your EB environment.

安装脚本在 /opt/elasticbeanstalk/hooks/appdeploy/post/ 文件夹 (文档) 存在于所有 EB 实例上.部署后将执行其中的任何 shell 脚本.放置在那里的 shell 脚本如下工作:

The setup script creates a file in the /opt/elasticbeanstalk/hooks/appdeploy/post/ folder (documentation) that lives on all EB instances. Any shell script in there will be executed post deployment. The shell script that is placed there works as follows:

1. 在celeryenv变量中，virutalenv环境存储在遵循 supervisord 符号的格式.这是逗号env 变量的分隔列表.
2. 然后脚本创建一个变量 celeryconf，其中包含配置文件作为字符串，其中包括先前解析的环境变量.
3. 这个变量然后被传送到一个名为 celeryd.conf 的文件中，一个celery 守护进程的 supervisord 配置文件.
4. 最后，将新创建的配置文件的路径添加到主 supervisord.conf 文件，如果它不存在.

1. In the celeryenv variable, the virutalenv environment is stored in a format that follows the supervisord notation. This is a comma separated list of env variables.
2. Then the script creates a variable celeryconf that contains the configuration file as a string, which includes the previously parsed env variables.
3. This variable is then piped into a file called celeryd.conf, a supervisord configuration file for the celery daemon.
4. Finally, the path to the newly created config file is added to the main supervisord.conf file, if it is not already there.

这是脚本的副本:

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash

      # Get django environment variables
      celeryenv=`cat /opt/python/current/env | tr '
' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
      celeryenv=${celeryenv%?}

      # Create celery configuraiton script
      celeryconf="[program:celeryd]
      ; Set full path to celery program if using virtualenv
      command=/opt/python/run/venv/bin/celery worker -A myappname --loglevel=INFO

      directory=/opt/python/current/app
      user=nobody
      numprocs=1
      stdout_logfile=/var/log/celery-worker.log
      stderr_logfile=/var/log/celery-worker.log
      autostart=true
      autorestart=true
      startsecs=10

      ; Need to wait for currently executing tasks to finish at shutdown.
      ; Increase this if you have very long running tasks.
      stopwaitsecs = 600

      ; When resorting to send SIGKILL to the program to terminate it
      ; send SIGKILL to its whole process group instead,
      ; taking care of its children as well.
      killasgroup=true

      ; if rabbitmq is supervised, set its priority higher
      ; so it starts first
      priority=998

      environment=$celeryenv"

      # Create the celery supervisord conf script
      echo "$celeryconf" | tee /opt/python/etc/celery.conf

      # Add configuration script to supervisord conf (if not there already)
      if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
          then
          echo "[include]" | tee -a /opt/python/etc/supervisord.conf
          echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
      fi

      # Reread the supervisord config
      supervisorctl -c /opt/python/etc/supervisord.conf reread

      # Update supervisord in cache without restarting all services
      supervisorctl -c /opt/python/etc/supervisord.conf update

      # Start/Restart celeryd through supervisord
      supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd

这篇关于您如何使用 AWS Elastic Beanstalk 运行工作线程?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！