工人“dyno"在 AWS Elastic Beanstalk 中 [英] Worker "dyno" in AWS Elastic Beanstalk
问题描述
Amazon Web Service 现在在其 Elastic Beanstalk 中有一个工作层.但是，它仍然让我们这些来自 Worker dyno 时代的人感到困惑.
Amazon Web Service has now a worker tiers in their Elastic Beanstalk. But, it nevertheless confuse us who come from the days of Worker dyno.
作为比较，在 Heroku 中，可以为 web 和 worker 分别配置两个 dynos(类似于处理器?).网络将适用于任何请求，并且通常会在 15 秒后超时.因此，如果您的请求持续时间更长，您的请求将只是超时，尽管本身并未终止.在这种情况下，您应该使用 worker 并且您的 Web dyno 应该每分钟(可能)访问端点数次以检查是否有任何结果要返回给用户.要制作工人或网络测功机，您只需要滑动滑块就可以了.有时，您可能需要一个 Procfile.但是没有什么花哨的，或者真正困难的，或者令人困惑的.
As a comparison, in Heroku, one can configure two dynos (something like processor?) each for web and worker. The web will work for any request, and will timeout normally at 15 secs. Thus, if you have a request that last more than that, your request will simply timed-out although not terminated per se. In that case, you should use worker and your web dyno should visit the endpoint several times per minutes (maybe) to check if there is any result to be brought back to the user. To make either worker or web dyno, what you need is just slide the slider and you are good to go. Sometimes, you may need a Procfile. But there is nothing fancy, or something really difficult, or confusing about it.
在 AWS EBS (Elastic Beanstalk) 中，从您点击 eb init 的第 1 天起，系统会询问您它是 Standard 还是 Worker.当你打标准时，似乎没有办法让它也成为工人.
In AWS EBS (Elastic Beanstalk), since day 1 you hit eb init, you will be asked whether it is a Standard or Worker. When you hit Standard, it seems there is no way to make it as worker as well.
在我们的情况下，worker 和标准网络位于一个应用程序下.那么，我们如何将 EBS 实例同时用于工人和标准.我们的工人正在使用 sidekiq 和 redis.请指出任何指导或帮助我们解决此问题.
In our situation, the worker and standard web is located under one application. So, how could we use an EBS instance both for worker and standard. Our worker is using sidekiq, and redis. Please, point to any guidance or help us in this matter.
推荐答案
AWS Elastic Beanstalk 有两种类型的环境 - Web 层和 Worker 层.
AWS Elastic Beanstalk has two types of Environments - Web tier and Worker tier.
Web 层环境适用于 Web 应用程序 - http/https 请求处理.您在负载均衡器后面获得一个或多个 EC2 实例.您可以根据需要获取其他资源，例如数据库.您可以选择您想要的平台，例如Ruby、Python、Java、Node.js、PHP、Docker.
Web tier environments are meant for web applications - http/https request processing. You get one or more EC2 instances behind a load balancer. You can get other resources like database per your requirement. You can choose the platform you wish e.g. Ruby, Python, Java, Node.js, PHP, Docker.
Worker 环境用于异步消息处理.创建工作环境时，您没有负载均衡器.您的所有 EC2 实例都在一个自动扩展组中.所有这些实例都在运行一个守护进程，它轮询单个 SQS 队列以获取消息.当守护程序从 SQS 队列中拉取消息时，守护程序会在 localhost:80 上发送 HTTP Post 请求.您可以配置端口，但重要的是守护进程将消息作为 HTTP 请求发布到本地主机上.您的工作应用程序实际上是一个 Web 应用程序，它接收 post 请求并处理消息.成功处理消息后，工作守护进程期望您在本地主机上运行的 Web 应用程序返回 HTTP 200 OK 响应.然后守护程序从 SQS 队列中删除消息.您可以像标准 Web 服务器应用程序一样为任何平台编写工作程序应用程序 - Ruby、Python、Java、Node.js、PHP、Docker.
Worker environments are meant for asynchronous message processing. When you create a worker environment you do not have a load balancer. All your EC2 instances are in an autoscaling group. All these instances are running a daemon which is polling a single SQS queue for messages. When a message is pulled by the daemon from the SQS queue, the daemon sends a HTTP Post request on localhost:80. You can configure the port but the important thing is that the daemon posts the message as an HTTP request on localhost. Your worker application is actually a web application that receives the post request and processes the message. After the message is successfully processed the worker daemon expects that your web application running on localhost returns a HTTP 200 OK response. The daemon then deletes the message from SQS queue. You can write your worker application for any platform just like standard web server applications - Ruby, Python, Java, Node.js, PHP, Docker.
根据我对您的用例的理解，我建议创建两个 Elastic Beanstalk 环境 - 一个标准环境和一个 Worker 环境.标准 Web 服务器接收 HTTP 请求并同步处理它们.此环境将相关数据放入 SQS 队列中.第二个环境是一个工作程序，在这个环境上运行的守护进程轮询这个 SQS 队列以获取消息.您的第二个环境是未对 Internet 开放的 Web 应用程序.工作守护进程将消息作为 HTTP 请求发布到您的工作环境.因此，您可以使用第二个工作环境异步处理长时间运行的工作负载.
Based on my understanding of your usecase I would recommend creating two Elastic Beanstalk environments - one Standard and one Worker environment. The Standard web server receives HTTP requests and processes them synchronously. This environment puts the relevant data in an SQS queue. The second environment is a worker and the daemon running on this environment polls this SQS queue for messages. Your second environment is a web application that is NOT open to the internet. The worker daemon posts the messages as HTTP requests to your worker environment. Thus you can process long running workloads asynchronously using this second worker environment.
在工作线程环境中，您可以使用自己的队列，或者 Elastic Beanstalk 可以为您生成一个队列.您可以根据需要配置消息可见性超时、http 连接等参数，也可以使用默认值.
With worker environments you can use your own queues or Elastic Beanstalk can generate a queue for you. You can configure parameters like message visibility timeout, http connections based on your requirements or you can use the defaults.
以下是一些可能对您有用的链接:
Below are some links that may be useful for you:
http://aws.amazon.com/blogs/aws/background-task-handling-for-aws-elastic-beanstalk/
https://stackoverflow.com/a/23942498/161628
这是否符合您的要求?如果您有其他问题，请告诉我.
Does this meet your requirements? Please let me know if you have further questions.
更新
您需要在两个地方上传源代码 - 一次用于工作环境，一次用于 Web 服务器环境.如果有人从头开始，那么他们可能有两个独立的代码库.但我认为在你的情况下，我认为在两个环境之间共享一个代码库应该是完全没问题的.假设您的 Web 请求到达/register"，那么您的应用程序中的 register() 方法可以将消息发布到 SQS 队列并通过 HTTP 请求完成.现在，您的工作环境将轮询 SQS 队列，并通过 localhost 上的 HTTP 将消息发布到 URL/async_register"，这将调用应用程序中的 async_register() 方法并执行异步处理.这两种方法可以存在于同一个源代码包中，可以由工作程序和 Web 服务器环境共享.Worker 和 Web 服务器采用的代码路径将不同，因此 Web 服务器环境将调用 register()，而Worker 环境将调用 async_register() 方法.
You need to upload your source code at two places - once for the worker environment and once for the web server environment. If someone was starting from scratch then they might have two separate code bases. But I think in your case I think it should be perfectly fine to have a single code base shared between the two environments. Suppose your web request arrives at '/register', then the register() method in your application can post messages to an SQS queue and be done with the HTTP request. Now your worker environment will poll the SQS queue and post messages over HTTP on localhost to the URL '/async_register' which will invoke a method async_register() in your application and do the asynchronous processing. These two methods can live in the same source code bundle which can be shared by both the worker and web server environment. The code path taken by worker and web server will be different so that web server environments will invoke register() and worker environments will invoke the async_register() method.
另一个警告是，本地主机上的工作守护进程发送的 HTTP 请求将包含一个 HTTP 标头 - "User-Agent": "aws-sqsd/1.1".在此处阅读更多信息.因此，在您的 Web 应用程序中，您可以有一个侦听器来在/register"上发布请求，并且根据此标头是否存在，您可以在内部调用 register() 或 async_register() 方法.
Another caveat is that HTTP requests sent by the worker daemon on localhost will contain an HTTP header - "User-Agent": "aws-sqsd/1.1". Read more here. So in your web application you can have a single listener to post requests on "/register" and depending on whether this header is present or not you invoke register() or async_register() methods internally.
另外我认为如果你想在两个环境之间共享代码库，你可以只在一个地方上传代码库.您的环境按逻辑分组为应用程序.因此，您可以拥有一个应用程序.您可以使用CreateApplicationVersion"API 调用将源代码上传到此应用程序.假设您上传标签为v1"的应用程序版本.您现在可以在同一应用程序下创建工作环境和 Web 服务器环境.创建环境时，您需要提供一个版本以部署到您的环境中.在这种情况下，您可以将 v1 部署到这两种环境.因此，您将为两种环境共享相同的源代码.当您有新版本v2"时.您上传此版本，然后在两个环境中执行更新，将其版本更改为v2".
Also I think if you want to share the code base between the two environments you can upload the code base at only one place. Your environments are logically grouped into applications. So you can have a single application. You upload your source code to this application using the "CreateApplicationVersion" API call. Suppose you upload an application version with label 'v1'. You can now create a worker environment and a web server environment under the same application. When you create an environment you need to provide a version to deploy to your enviroment. In this case you can deploy v1 to both environments. So you will be sharing the same source code for both environments. When you have a new version "v2". You upload this version and then perform an update on both environments changing their version to "v2".
相同版本的源代码可以部署到两种环境中.它们将在不同的 EC2 实例上运行，因为一个环境专用于响应 Web 请求，而一个环境专用于响应异步 Web 请求(worker).
The same version of the source code can be deployed to both environments. They will be running on different EC2 instances because one environment is dedicated for responding to web requests and one environment is dedicated for responding to asynchronous web requests (worker).
这篇关于工人“dyno"在 AWS Elastic Beanstalk 中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
