调试长时间运行的PHP脚本 [英] debugging long running PHP script

问题描述

我有php脚本作为cron作业运行，广泛使用第三方代码。脚本本身有几千个LOC。基本上它是数据导入/处理脚本。 （JSON到MySQL，但它也使得很多HTTP调用和一些SOAP）。

现在，性能随着时间的推移而降级。当测试几个记录（约100），性能是确定，它是在10-20分钟内完成。当运行整个进口（约1600条记录）时，一个记录的平均进口时间稳步增长，整体需要24小时以上，至少比预期的要长5倍。

记忆似乎不是一个问题，使用量应该会增长，没有意想不到的高峰。

所以，我需要调试它来找到瓶颈。脚本，底层代码库，php本身，数据库，os或网络部分可能会有一些问题。我怀疑现在有某种缓存在一个接近100％错失率的行为不佳的地方。

我不能使用XDebug，配置文件增长太快，不能

所以问题是：如何调试这种脚本？

PHP版本：5.4.41
操作系统：Debian 7.8
如果需要，我可以拥有root权限，并安装工具。但是它的生产服务器和理想的调试不应该太中断。

解决方案

剖析工具：

目前公开测试版的

内存消费可疑：

内存似乎不是一个问题，使用增长应该是没有意想不到的高峰。

越来越多的内存使用实际上听起来很可疑！如果当前数据集不依赖于导入的所有以前的数据集，那么生长的内存最可能意味着所有导入的数据集都保存在内存中，这是坏的。 PHP也可能经常尝试垃圾收集，只是为了找出没有什么可以从内存中删除。特别是长时间运行的CLI任务会受到影响，所以请务必阅读博客发现了这种行为。

I have php script running as a cron job, extensively using third party code. Script itself has a few thousands LOC. Basically it's the data import / treatment script. (JSON to MySQL, but it also makes a lot of HTTP calls and some SOAP).

Now, performance is downgrading with the time. When testing with a few records (around 100), performance is ok, it is done in a 10-20 minutes. When running whole import (about 1600 records), mean time of import of one record grows steadily, and whole thing takes more than 24 hours, so at least 5 times longer than expected.

Memory seems not to be a problem, usage growing as it should, without unexpected peaks.

So, I need to debug it to find the bottleneck. It can be some problem with the script, underlying code base, php itself, database, os or network part. I am suspecting for now some kind of caching somewhere which is not behaving well with a near 100 % miss ratio.

I cannot use XDebug, profile file grows too fast to be treatable.

So question is: how can I debug this kind of script?

PHP version: 5.4.41 OS: Debian 7.8 I can have root privileges if necessary, and install the tools. But it's the production server and ideally debugging should not be too disrupting.

解决方案

Profiling tool:

There is a PHP profiling tool called Blackfire which is currently in public beta. There is specific documentation on how to profile CLI applications. Once you collected the profile you can analyze the application control flow with time measurements in a nice UI:

Memory consumption suspicious:

Memory seems not to be a problem, usage growing as it should, without unexpected peaks.

A growing memory usage actually sounds suspicious! If the current dataset does not depend on all previous datasets of the import, then a growing memory most probably means, that all imported datasets are kept in memory, which is bad. PHP may also frequently try to garbage collect, just to find out that there is nothing to remove from memory. Especially long running CLI tasks are affected, so be sure to read the blog post that discovered the behavior.

这篇关于调试长时间运行的PHP脚本的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！