降低Perl内存使用量的提示 [英] Tips for keeping Perl memory usage low

问题描述

在Perl脚本中保持较低的内存使用量的一些好的技巧是什么?我对学习如何使依赖Perl程序的系统的内存占用量尽可能低感到兴趣.我知道Perl在内存使用方面并不是很好，但是我想知道是否有任何改进技巧.

因此，您可以怎么做才能使用更少的内存来保留Perl脚本.我对任何建议都感兴趣，无论它们是编写代码的实际技巧，还是有关如何以不同方式编译Perl的技巧.

编辑赏金: 我有一个perl程序，可作为网络应用程序的服务器.当前连接到它的每个客户端都有它自己的子进程.我也使用线程代替了分叉，但是我无法确定使用线程代替分叉实际上是否更有效地利用内存.

我想再次尝试使用线程而不是fork.我相信从理论上讲，它应该可以节省内存使用量.在这方面，我有几个问题:

1. 在Perl中创建的线程是否阻止复制Perl模块库 进入每个线程的内存?
2. 线程(使用线程)是最有效的方法(或唯一的方法) ) Perl中创建线程的方法?
3. 在线程中，我可以指定stack_size参数，具体是什么 指定此值时应考虑什么，以及它如何影响 内存使用情况?

对于Perl/Linux中的线程，最可靠的方法是确定每个线程的实际内存使用情况?

解决方案

您遇到什么问题，大"对您意味着什么?我有朋友，您需要将200 Gb文件加载到内存中，所以他们的小技巧与预算购物者有很大的不同，因为预算最少的VM切片需要250 MB的RAM(真的吗?我的手机还可以提供更多服务). >

通常，Perl会保留您使用的任何内存，即使它没有使用它也是如此.实现一个方向的优化，例如内存可能会对速度产生负面影响，例如速度.

这不是一个完整的列表(编程Perl 中有更多列表):

☹使用Perl内存分析工具来帮助您查找问题区域.请参见对Perl程序中的堆内存使用情况进行分析和内存映射文件，而不是对其进行压缩

☹如果您需要创建大数据结构，请考虑类似 DBM :: Deep 或其他存储引擎将其大部分保留在RAM之外并保留在磁盘上，直到需要时为止.

☹不要让别人使用您的程序.每当我这样做时，我都会将内存占用量减少约100％.它还减少了支持请求.

☹通过引用传递大量的文本和较大的聚合，因此您无需进行复制，因此可以将相同的信息存储两次.如果由于要更改某些内容而必须复制它，则可能会卡住.这既可以作为子例程参数，也可以作为子例程返回值:

 call_some_sub( \$big_text, \@long_array );
 sub call_some_sub {
      my( $text_ref, $array_ref ) = @_;
      ...
      return \%hash;
      }

☹跟踪模块中的内存泄漏.在意识到模块没有释放内存.我在模块的RT队列中找到了补丁，将其应用并解决了问题.

☹如果您需要一次处理大量数据，但又不想占用持久的内存，则将工作分担给子进程.子进程在工作时仅具有内存占用量.当您得到答案时，子进程将关闭并释放其内存.同样，诸如 Gearman 之类的工作分配系统可以在机器之间分散工作.

☹将递归解转换为迭代解. Perl没有尾部递归优化，因此每个新调用都会添加到调用堆栈中.您可以使用 goto 或模块自己动手优化尾巴问题，但这很多依靠您可能不需要的技术进行工作.

☹他使用6 Gb还是仅使用5 Gb?好吧，说实话，在所有这些激动中，我有点迷失了自己.但是因为这是Perl(世界上最强大的语言)，并且会清除内存，所以您必须问自己一个问题:我感到幸运吗?好吧，朋克吗?

还有更多，但是现在来弄清楚这些是什么还为时过早.我在精通Perl 和threads (use threads) the most efficient way (or the only) way to create threads in Perl?

In threads, I can specify a stack_size paramater, what specifically should I consider when specifying this value, and how does it impact memory usage?

With threads in Perl/Linux, what is the most reliable method to determine the actual memory usage on a per-thread basis?

解决方案

What sort of problem are you running into, and what does "large" mean to you? I have friends you need to load 200 Gb files into memory, so their idea of good tips is a lot different than the budget shopper for minimal VM slices suffering with 250 Mb of RAM (really? My phone has more than that).

In general, Perl holds on to any memory you use, even if it's not using it. Realize that optimizing in one direction, e.g. memory, might negatively impact another, such as speed.

This is not a comprehensive list (and there's more in Programming Perl):

☹ Use Perl memory profiling tools to help you find problem areas. See Profiling heap memory usage on perl programs and How to find the amount of physical memory occupied by a hash in Perl?

☹ Use lexical variables with the smallest scope possible to allow Perl to re-use that memory when you don't need it.

☹ Avoid creating big temporary structures. For instance, reading a file with a foreach reads all the input at once. If you only need it line-by-line, use while.

 foreach ( <FILE> ) { ... } # list context, all at once 
 while( <FILE> ) { ... } # scalar context, line by line

☹ You might not even need to have the file in memory. Memory-map files instead of slurping them

☹ If you need to create big data structures, consider something like DBM::Deep or other storage engines to keep most of it out of RAM and on disk until you need it.

☹ Don't let people use your program. Whenever I've done that, I've reduced the memory footprint by about 100%. It also cuts down on support requests.

☹ Pass large chunks of text and large aggregates by reference so you don't make a copy, thus storing the same information twice. If you have to copy it because you want to change something, you might be stuck. This goes both ways as subroutine arguments and subroutine return values:

 call_some_sub( \$big_text, \@long_array );
 sub call_some_sub {
      my( $text_ref, $array_ref ) = @_;
      ...
      return \%hash;
      }

☹ Track down memory leaks in modules. I had big problems with an application until I realized that a module wasn't releasing memory. I found a patch in the module's RT queue, applied it, and solved the problem.

☹ If you need to handle a big chunk of data once but don't want the persistent memory footprint, offload the work to a child process. The child process only has the memory footprint while it's working. When you get the answer, the child process shuts down and releases it memory. Similarly, work distribution systems, such as Gearman, can spread work out among machines.

☹ Turn recursive solutions into iterative ones. Perl doesn't have tail recursion optimization, so every new call adds to the call stack. You can optimize the tail problem yourself with tricks with goto or a module, but that's a lot of work to hang onto a technique that you probably don't need.

☹ Did he use 6 Gb or only five? Well, to tell you the truth, in all this excitement I kind of lost track myself. But being as this is Perl, the most powerful language in the world, and would blow your memory clean off, you've got to ask yourself one question: Do I feel lucky? Well, do ya, punk?

There are many more, but it's too early in the morning to figure out what those are. I cover some in Mastering Perl and Effective Perl Programming.

这篇关于降低Perl内存使用量的提示的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！