使用PHP在大型数据库上执行Cron作业，需要建议 [英] Cron job on a large database with PHP, suggestions needed

问题描述

我听说过cron作业，并且不认为很难真正创建它，但是我担心它如何在大型脚本中工作.

I've heard about cron job and don't think the actual creation of it will be that hard to make but I've some concerns about how this will work with a large script.

在我的项目中，不要过多地偏离主题，我会坚持我所处情况的基础知识.我需要制作一个脚本，该脚本每天对远程网站上的数据执行CURL提取，并使用它来更新我网站上每个重要成员的数据库.简而言之，此时大约需要执行脚本的1000倍，但是随着时间的流逝它会更大.

Without going too much off-topic on my project i will stick with the basics about my situation. I need to make a script that every day performs a CURL fetch for data on a remote website and updates an database for each featured member on my website with it. In short, it's approximatively at this time 1000 times the script need to be executed but it will be a larger number as times goes by.

您可以猜测，这将需要很长的时间才能执行，因此我担心执行如何以不会崩溃的方式工作.

As you can guess, this will take a long time to preform so i'm worried about how the execution will work in a manner of not crashing in the middle of it.

我的第一个想法是也许将用户分成几组，每次都在少量用户上执行，但是不知道这是如何可管理的(当我得到某种形式的确认时，将进一步阅读该主题.在此).

My first thought was to perhaps split the users into groups and make the executions on a small amount of users each time but don't know how this is manageable ( will read on further about the topic when i got some form of confirmation on this).

所以，对我的问题.您认为我有什么办法可以实现这一目标，或者您对如何使其有效运行有任何建议吗?我能得到的所有帮助都受到赞赏.感谢您的时间.

So, to my question. Do you think there is any way for me to make this happen and do you perhaps have any suggestions on how to make this to work efficiently? All help i can get is appreciated. Thank you for your time.

推荐答案

使用php和mysql的更大的cron-jobs必须是零碎的，因为没有办法让您很好"地使用它们(降低它们的操作系统优先级).即使您喜欢该脚本，也无需担心就可以执行mysql请求.

bigger cron-jobs with php and mysql needs to be fragmented, since there is no way for you to 'nice' them, (reduce their os priority). Even if you nice the script, the mysql-requests will be executed without this concern.

根据您的描述，有两个方面需要考虑:

From what you're describing there's two aspects to consider:

• 网络带宽拥塞
• 数据库吞吐量拥挤

我建议使用一个分散的解决方案，在该解决方案中，您可以更频繁地从cron调用脚本，并让脚本仅执行少量的总任务.如果I/O带宽或CPU使用率超出可能影响访问者响应时间的任何限制，则应进一步取消该工作(推迟到下一次运行).

I'd recommend a fragmented solution where you call your script from cron more often, and let the script execute only a small amount of the total job. The job should further be canceled (postponed to next run) if i/o-bandwith or cpu-usage is above any limit that may affect response-time to visitors.

致谢，/t

这篇关于使用PHP在大型数据库上执行Cron作业，需要建议的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！