在 Perl 中，如何向操作系统释放内存? [英] In Perl, how can I release memory to the operating system?
问题描述
我在 Perl 中遇到了一些内存问题.当我填满一个大哈希时，我无法将内存释放回操作系统.当我对标量执行相同操作并使用 undef 时，它会将内存返还给操作系统.
这是我写的一个测试程序.
#!/usr/bin/perl###### 记忆测试######## 使用命令使用 Number::Bytes::Human qw(format_bytes);使用数据::倾销者；使用 Devel::Size qw(size total_size);## 创建变量我的 $share_var;我的 %share_hash;我的 $type_hash = 1;我的 $type_scalar = 1;## 启动主循环而(真){&Memory_Check();打印按回车键(添加到内存):"；<>;&Up_Mem(100_000);&Memory_Check();打印按回车键(将变量设置为空):"；<>;$share_var = "";$share_hash = ();&Memory_Check();打印按回车键(清除数据):"；<>;&Clean_Data();&Memory_Check();打印按回车键(重新开始):"；<>;}出口;####向上记忆子 Up_Mem {我的 $total_loops = shift;我的 $n = 1;print "将数据添加到共享变量 $total_loops 次
";直到 ($n > $total_loops) {如果($type_hash){$share_hash{$n} = 'X' x 1111;}如果($type_scalar){$share_var .= 'X' x 1111;}$n += 1;}打印完成添加数据
"；}####清理数据子 Clean_Data {打印清理数据
"；如果($type_hash){## 修复哈希的方法(尝试我能想到的一切！我的 $n = 1;我的 $total_loops = 100_000;直到 ($n > $total_loops) {undef $share_hash{$n};$n += 1;}%share_hash = ();$share_hash = ();undef $share_hash;undef %share_hash;}如果($type_scalar){undef $share_var;}}#### 检查内存使用情况子内存_检查{##从shell获取当前内存我的@mem = `ps aux |grep "$$"`;my($results) = grep !/grep/, @mem;## 解析来自 Shell 的数据chomp $results;$results =~ s/^w*s*d*s*d*.d*s*d*.d*s*//g;$results =~ s/pts.*//g;我的 ($vsz,$rss) = split(/s+/,$results);## 将数字格式化为人类可读的我的 $h = Number::Bytes::Human->new();我的 $virt = $h->format($vsz);我的 $h = Number::Bytes::Human->new();我的 $res = $h->format($rss);打印当前内存使用情况:Virt:$virt RES:$res
"；如果($type_hash){我的 $total_size = total_size(\%share_hash);我的@arr_c = 键 %share_hash;打印哈希长度:".($#arr_c + 1) ." 哈希内存总大小:$total_size
";}如果($type_scalar){我的 $total_size = total_size($share_var);打印标量长度:".长度($share_var) ." 标量内存总大小:$total_size
";}}输出:
<前>./Memory_Undef_Simple.cgi当前内存使用:Virt:6.9K RES:2.7K哈希长度:0 哈希内存总大小:92标量长度:0 标量内存总大小:12按 Enter(添加到内存中):向共享变量添加数据 100000 次完成添加数据当前内存使用:Virt:228K RES:224K哈希长度:100000 哈希内存总大小:116813243标量长度:111100000 标量内存总大小:111100028按 Enter(将变量设置为空):当前内存使用:Virt:228K RES:224K哈希长度:100000 哈希内存总大小:116813243标量长度:0 标量内存总大小:111100028按 Enter(清除数据):清理数据当前内存使用:Virt:139K RES:135K哈希长度:0 哈希内存总大小:92标量长度:0 标量内存总大小:24按 Enter(重新开始):所以你可以看到内存减少了，但它只减少了标量的大小.任何想法如何释放哈希的内存?
还有 Devel::Size 显示哈希即使程序仍然使用139K，也只占用92个字节.
一般来说，是的，UNIX 上的内存管理就是这样工作的.如果您使用带有最新 glibc 的 Linux，并且正在使用该 malloc，您可以将释放的内存返回给操作系统.不过，我不确定 Perl 是否会这样做.
如果你想处理大型数据集，不要将整个数据加载到内存中，使用像 BerkeleyDB 这样的东西:
https://metacpan.org/pod/BerkeleyDB
示例代码，逐字窃取:
使用严格；使用 BerkeleyDB ；我的 $filename = "fruit" ;取消链接 $filename ;绑定我的 %h，BerkeleyDB::Hash"，-文件名 =>$文件名，-标志 =>数据库_创建或者死 "无法打开文件 $filename: $! $BerkeleyDB::Error
" ;# 在文件中添加几个键/值对$h{apple} = "红色" ;$h{orange} = "橙色" ;$h{banana} = "黄色" ;$h{番茄} = "红色" ;# 检查密钥是否存在如果 $h{banana} ，则打印香蕉存在
"；# 删除一个键/值对.删除 $h{apple} ;# 打印文件内容while (my ($k, $v) = 每个 %h){ 打印 "$k -> $v
" }解开 %h ;(好吧，不是逐字逐句的.他们对 use vars 的使用是......遗留......)
您可以通过这种方式在散列中存储千兆字节的数据，而且您将只使用一点点内存.(基本上，无论 BDB 的寻呼机决定保留在内存中；这是可控的.)
I am having some problems with memory in Perl. When I fill up a big hash, I can not get the memory to be released back to the OS. When I do the same with a scalar and use undef, it will give the memory back to the OS.
Here is a test program I wrote.
#!/usr/bin/perl
###### Memory test
######
## Use Commands
use Number::Bytes::Human qw(format_bytes);
use Data::Dumper;
use Devel::Size qw(size total_size);
## Create Varable
my $share_var;
my %share_hash;
my $type_hash = 1;
my $type_scalar = 1;
## Start Main Loop
while (true) {
&Memory_Check();
print "Hit Enter (add to memory): "; <>;
&Up_Mem(100_000);
&Memory_Check();
print "Hit Enter (Set Varable to nothing): "; <>;
$share_var = "";
$share_hash = ();
&Memory_Check();
print "Hit Enter (clean data): "; <>;
&Clean_Data();
&Memory_Check();
print "Hit Enter (start over): "; <>;
}
exit;
#### Up Memory
sub Up_Mem {
my $total_loops = shift;
my $n = 1;
print "Adding data to shared varable $total_loops times
";
until ($n > $total_loops) {
if ($type_hash) {
$share_hash{$n} = 'X' x 1111;
}
if ($type_scalar) {
$share_var .= 'X' x 1111;
}
$n += 1;
}
print "Done Adding Data
";
}
#### Clean up Data
sub Clean_Data {
print "Clean Up Data
";
if ($type_hash) {
## Method to fix hash (Trying Everything i can think of!
my $n = 1;
my $total_loops = 100_000;
until ($n > $total_loops) {
undef $share_hash{$n};
$n += 1;
}
%share_hash = ();
$share_hash = ();
undef $share_hash;
undef %share_hash;
}
if ($type_scalar) {
undef $share_var;
}
}
#### Check Memory Usage
sub Memory_Check {
## Get current memory from shell
my @mem = `ps aux | grep "$$"`;
my($results) = grep !/grep/, @mem;
## Parse Data from Shell
chomp $results;
$results =~ s/^w*s*d*s*d*.d*s*d*.d*s*//g; $results =~ s/pts.*//g;
my ($vsz,$rss) = split(/s+/,$results);
## Format Numbers to Human Readable
my $h = Number::Bytes::Human->new();
my $virt = $h->format($vsz);
my $h = Number::Bytes::Human->new();
my $res = $h->format($rss);
print "Current Memory Usage: Virt: $virt RES: $res
";
if ($type_hash) {
my $total_size = total_size(\%share_hash);
my @arr_c = keys %share_hash;
print "Length of Hash: " . ($#arr_c + 1) . " Hash Mem Total Size: $total_size
";
}
if ($type_scalar) {
my $total_size = total_size($share_var);
print "Length of Scalar: " . length($share_var) . " Scalar Mem Total Size: $total_size
";
}
}
OUTPUT:
./Memory_Undef_Simple.cgi Current Memory Usage: Virt: 6.9K RES: 2.7K Length of Hash: 0 Hash Mem Total Size: 92 Length of Scalar: 0 Scalar Mem Total Size: 12 Hit Enter (add to memory): Adding data to shared varable 100000 times Done Adding Data Current Memory Usage: Virt: 228K RES: 224K Length of Hash: 100000 Hash Mem Total Size: 116813243 Length of Scalar: 111100000 Scalar Mem Total Size: 111100028 Hit Enter (Set Varable to nothing): Current Memory Usage: Virt: 228K RES: 224K Length of Hash: 100000 Hash Mem Total Size: 116813243 Length of Scalar: 0 Scalar Mem Total Size: 111100028 Hit Enter (clean data): Clean Up Data Current Memory Usage: Virt: 139K RES: 135K Length of Hash: 0 Hash Mem Total Size: 92 Length of Scalar: 0 Scalar Mem Total Size: 24 Hit Enter (start over):
So as you can see the memory goes down, but it only goes down the size of the scalar. Any ideas how to free the memory of the hash?
Also Devel::Size shows the hash is only taking up 92 bytes even though the program still is using 139K.
Generally, yeah, that's how memory management on UNIX works. If you are using Linux with a recent glibc, and are using that malloc, you can return free'd memory to the OS. I am not sure Perl does this, though.
If you want to work with large datasets, don't load the whole thing into memory, use something like BerkeleyDB:
https://metacpan.org/pod/BerkeleyDB
Example code, stolen verbatim:
use strict ;
use BerkeleyDB ;
my $filename = "fruit" ;
unlink $filename ;
tie my %h, "BerkeleyDB::Hash",
-Filename => $filename,
-Flags => DB_CREATE
or die "Cannot open file $filename: $! $BerkeleyDB::Error
" ;
# Add a few key/value pairs to the file
$h{apple} = "red" ;
$h{orange} = "orange" ;
$h{banana} = "yellow" ;
$h{tomato} = "red" ;
# Check for existence of a key
print "Banana Exists
" if $h{banana} ;
# Delete a key/value pair.
delete $h{apple} ;
# print the contents of the file
while (my ($k, $v) = each %h)
{ print "$k -> $v
" }
untie %h ;
(OK, not verbatim. Their use of use vars is ... legacy ...)
You can store gigabytes of data in a hash this way, and you will only use a tiny bit of memory. (Basically, whatever BDB's pager decides to keep in memory; this is controllable.)
这篇关于在 Perl 中，如何向操作系统释放内存?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
