从Perl获取堆栈跟踪“内存不足".错误 [英] Getting stack trace from Perl "Out of memory" error

问题描述

tl; dr:当Perl httpd进程内存不足时，如何转储perl堆栈跟踪.

tl;dr: How to dump a perl stack trace when a Perl httpd process runs out of memory.

我们有一个mod_perl 2服务器，Perl 5.8.8，RHEL 5.6，Linux 2.6.18.

We've got a mod_perl 2 server, Perl 5.8.8, RHEL 5.6, Linux 2.6.18.

非常偶尔且不可预测的是，子httpd进程开始以惊人的速度耗尽所有可用内存.我们至少使用了BSD :: Resource :: setrlimit(RLIMIT_VMEM，...)，以便在关闭服务器之前，该进程以内存不足"而终止.

Very occasionally and unpredictably, a child httpd process starts using up all available memory at an alarming rate. We've at least used BSD::Resource::setrlimit(RLIMIT_VMEM, ...), so that the process dies with "Out of memory" before bringing down the server.

我们不知道代码在哪里发生，并且如果不进行数小时的负载测试就很难重现.

We don't know where in the code this is happening, and it's difficult to reproduce without hours of load testing.

我们真正想要的是一种在进程内存不足之前获取 Perl堆栈跟踪的方法，因此我们知道是什么代码导致了这一点.不幸的是，内存不足"是一个无法解决的错误.

What we'd really like is a way to get a Perl stack trace just before the process runs out of memory, so we know what code is causing this. Unfortunately, "Out of memory" is an untrappable error.

以下是我正在考虑的选项，每个选项都有其缺点:

Here are the options I'm considering, each with their drawbacks:

1)使用 $ ^ M紧急存储池.要求我们使用-DPERL_EMERGENCY_SBRK和-Dusemymalloc重新编译perl.

1) Use the $^M emergency memory pool. Requires us to recompile perl with -DPERL_EMERGENCY_SBRK and -Dusemymalloc.

2)输入大量的日志语句，然后分析日志以查看流程在何处停止.

2) Put in tons of log statements, then analyze the logs to see where the process is stopping short.

3)编写一个外部脚本，该脚本不断扫描httpd进程池，如果发现其中一个使用大量内存，则向其发送USR2信号(我们已安排该信号转储堆栈跟踪).

3) Write an external script that constantly scans the pool of httpd processes, and if it sees one using a lot of memory, sends it a USR2 signal (which we've arranged to dump a stack trace).

4)以某种方式让进程连续监视自己的内存，并在内存变高但在内存不足"错误发生之前转储堆栈跟踪.

4) Somehow have the process monitor its own memory continuously, and dump a stack trace when memory gets high but before the "Out of Memory" error.

谢谢！

乔恩

推荐答案

您可以尝试使用LD_PRELOAD加载自定义版本的malloc/free.不必重新链接或重新编译任何内容.只要将LD_PRELOAD设置为具有与malloc相同接口的.so，在运行可执行文件时，将加载此预加载版本的malloc而不是正常的系统malloc.例如，这是使用 efence 检测内存错误的常用策略.

You might try using LD_PRELOAD to load a custom version of malloc/free. It should not be necessary to relink or recompile anything. Just set LD_PRELOAD to have a .so that has the same interface as malloc, and this preloaded version of malloc will be loaded instead of the normal system malloc when you run your executable. For instance, this is a common strategy for detecting memory errors with efence.

我认为efence在这里不适合您，因为这是用于检测内存覆盖，而不是调试内存不足(我不知道它对OOM的作用).我认为您可能想查看 failmalloc .我从来没有使用过它，但听起来它可能可以满足您的要求(我只是浏览了首页).

I don't think efence will work for you here as that is for detecting memory overwrites, not debugging out of memory (I don't know what it does on OOM). I think what you might want to check out failmalloc. I have never used this one but it sounds like it might do what you want (I only skimmed the front page).

这篇关于从Perl获取堆栈跟踪“内存不足".错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！