PHP 中的异步处理或消息队列 (CakePHP) [英] Asynchronous processing or message queues in PHP (CakePHP)

问题描述

我正在用 CakePHP 构建一个网站，用于处理通过 XML-RPC API 和 Web 前端上传的文件.文件需要被 ClamAV 扫描，需要生成缩略图等等.所有需要一些时间的资源密集型工作，用户不应等待.所以，我正在研究使用 PHP 的异步处理，特别是 CakePHP.

I am building a website in CakePHP that processes files uploaded though an XML-RPC API and though a web frontend. Files need to be scanned by ClamAV, thumbnails need to be generated, et cetera. All resource intensive work that takes some time for which the user should not have to wait. So, I am looking into asynchronous processing with PHP in general and CakePHP in particular.

我发现了 MultiTask 插件 对于看起来很有希望的 CakePHP.我还遇到了各种消息队列实现，例如 dropr 和 beanstalkd.当然，我还需要某种后台进程，可能是使用某种 Cake Shell 实现的.我看到 MultiTask 使用 PHP_Fork 来实现多线程 PHP 守护进程.

I came across the MultiTask plugin for CakePHP that looks promising. I also came across various message queue implementations such as dropr and beanstalkd. Of course, I will also need some kind of background process, probably implemented using a Cake Shell of some kind. I saw MultiTask using PHP_Fork to implement a multithreaded PHP daemon.

我需要一些关于如何以最佳方式将所有这些部分组合在一起的建议.

I need some advice on how to fit all these pieces together in the best way.

• 用 PHP 编写一个长时间运行的守护进程是个好主意吗?我应该注意什么?
• 外部消息队列实现的优势是什么?MultiTask 插件不使用外部消息队列.它使用 MySQL 表来存储任务.
• 我应该使用什么消息队列?滴滴?豆茎?还有什么?
• 我应该如何实现后端处理器?分叉 PHP 守护程序是一个好主意还是只是自找麻烦?

我目前的计划是使用 MultiTask 插件或编辑它以使用 beanstald 而不是它自己的 MySQL 表实现.队列中的作业可以简单地由任务名称和参数数组组成.PHP 守护进程将监视传入的作业并将它们传递给它的子线程之一.将使用给定的参数简单地执行 CakePHP 任务.

My current plan is either to use the MultiTask plugin or to edit it to use beanstald instead of it's own MySQL table implementation. Jobs in the queue can simply consist of a task name and an array of parameters. The PHP daemon would watch for incoming jobs and pass them out to one of it's child threads. The would simply execute the CakePHP Task with the given parameters.

对此有任何意见、建议、评论、陷阱或火焰吗?

Any opinion, advice, comments, gotchas or flames on this?

推荐答案

我使用 BeanstalkD 取得了很好的效果 和一个用 PHP 编写的后端来检索作业，然后对它们采取行动.我将实际运行的作业包装在 bash 脚本中，即使它退出也能继续运行(除非我执行exit(UNIQNUM);"，当脚本检查它并实际退出时).这样，重新启动的 PHP 脚本会清除可能已使用的所有内存，并且可以每运行 25/50/100 个作业重新启动.

I've had excellent results with BeanstalkD and a back-end written in PHP to retrieve jobs and then act on them. I wrapped the actual job-running in a bash-script to keep running if even if it exited (unless I do a 'exit(UNIQNUM);', when the script checks it and will actually exit). In that way, the restarted PHP script clears down any memory that may have been used, and can start afresh every 25/50/100 jobs it runs.

使用它的几个优点是您可以在 BeanstalkD 作业中设置优先级和延迟 - 以较低的优先级运行它，但不要启动 10 秒".我还曾在某个时间将许多作业排队(现在运行，5 秒后运行，30 秒后再次运行).

A couple of the advantages of using it is that you can set priorities and delays into a BeanstalkD job - "run this at a lower priority, but don't start for 10 seconds". I've also queued a number of jobs up at the some time (run this now, in 5 seconds and again after 30 secs).

通过适当的网络配置(并在网络其余部分的可访问 IP 地址上运行它)，您还可以在一台服务器上运行 beanstalkd 守护进程，并从许多其他机器轮询它，因此如果有如果正在生成大量任务，则可以在服务器之间拆分工作.如果需要在特定机器上运行一组特定任务，我创建了一个管"，它是该机器的主机名，如果不是全局的，它在我们的集群中应该是唯一的(对于文件上传很有用).我发现它非常适合调整图像大小，通常在引用它的网页本身引用它将到达的 URL 之前将完成的较小图像返回到文件系统.

With the appropriate network configuration (and running it on an accessible IP address to the rest of your network), you can also run a beanstalkd deamon on one server, and have it polled from a number of other machines, so if there are a large number of tasks being generated, the work can be split off between servers. If a particular set of tasks needs to be run on a particular machine, I've created a 'tube' which is that machine's hostname, which should be unique within our cluster, if not globally (useful for file uploads). I found it worked perfectly for image resizing, often returning the finished smaller images to the file system before the webpage itself that would refer to it would refer to the URL it would be arriving at.

我实际上即将开始为我的博客撰写关于这个主题的系列文章(包括一些我已经通过数百万个实时请求推送的代码技术) - 我的 URL 是从我的 用户个人资料在这里，在 Stackoverflow 上.

I'm actually about to start writing a series of articles on this very subject for my blog (including some techniques for code that I've already pushed several million live requests through) - My URL is linked from my user profile here, on Stackoverflow.

(我写了一篇关于 Beanstalkd 和排队主题的系列文章工作)

(I've written a series of articles on the subject of Beanstalkd and queuing of jobs)

这篇关于PHP 中的异步处理或消息队列 (CakePHP)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！