使用Perl访问BerkeleyDB的正确方法是什么？ [英] What is the proper way to access BerkeleyDB with Perl?
问题描述
我使用BerkeleyDB时遇到了一些问题。我有相同的代码的多个实例指向单个存储库的DB文件，一切运行良好5-32小时，然后突然有一个死锁。命令提示在执行db_get或db_put或游标创建调用之前立即停止。所以我只是要求正确的方式来处理这些调用。这是我的总体布局：
这是环境和DB的创建方式：
my $ env = new BerkeleyDB :: Env（
-Home =>$ dbFolder\\，
-Flags => DB_CREATE | DB_INIT_CDB | DB_INIT_MPOOL）
或die无法打开环境：$ BerkeleyDB :: Error\\\
;
my $ unsortedHash = BerkeleyDB :: Hash-> new（
-Filename =>$ dbFolder / Unsorted.db，
-Flags => DB_CREATE，
-Env => $ env
）或死无法创建：$ !, $ BerkeleyDB :: Error.\\\
;
此代码的单个实例运行，转到网站并保存要由另一个实例解析的URL （我有标志设置，以便每当锁定时锁定每个DB）：
$ lk = $ unsortedHash-> cds_lock（）;
while（@urlsToAdd）{
my $ currUrl = shift @urlsToAdd;
$ unsortedHash-> db_put（$ currUrl，'0'）;
}
$ lk-> cds_unlock（）;
定期检查某些数量的项目是否未排序：
$ refer = $ unsortedHash-> db_stat（）;
$ elements = $ refer-> {'hash_ndata'};
在将任何元素添加到任何数据库之前，它首先检查所有数据库以查看该元素是否已经存在：
if（$ unsortedHash-> db_get（$ search，$ value）== 0）{
$ value =1：$ value;
} elsif（$ badHash-> db_get（$ search，$ value）== 0）{
$ value =2：$ value;
....
下一个代码后面，在平行下。首先，它获得未排序的下一个项目（不具有繁忙值'1'），然后将值设置为忙'1'，然后使用它，然后将数据库条目完全移动到另一个数据库（它是从未分类中移除并存储在另一个数据库中）：
my $ pageUrl ='';
my $ busy ='1';
my $ curs;
my $ lk = $ unsortedHash-> cds_lock（）; #lock，将状态更改为1，unlock
##########从UNSORTED HASH获取元素######
while（1）{
$ busy ='1';
$ curs = $ unsortedHash-> db_cursor（）;
while（$ busy）{
$ curs-> c_get（$ pageUrl，$ busy，DB_NEXT）;
print$ pageUrl：$ busy：\\\
;
if（$ pageUrl eq''）{
$ busy = 0;
}
}
$ curs-> c_close（）;
$ curs = undef;
if（$ pageUrl eq''）{
printDatabase empty。Sleeping ... \\\
;
$ lk-> cds_unlock（）;
sleep（30）;
$ lk = $ unsortedHash-> cds_lock（）;
} else {
last;
}
}
####### MAKE THE ELEMENT'BUSY'and DOWNLOAD IT
$ unsortedHash-> ; db_put（$ pageUrl，'1'）;
$ lk-> cds_unlock（）;
$ lk = undef;
在其他地方，如果我在任何数据库上调用db_put或db_del，锁定像这样：
打印\\\？） rel =nofollow> Berkeley DB程序员参考指南。
\\\
Bad.\\\
\\\
;
$ lk = $ badHash-> cds_lock（）;
$ badHash-> db_put（$ pageUrl，'0'）;
$ unsortedHash-> db_del（$ pageUrl）;
$ lk-> cds_unlock（）;
$ lk = undef;但是，我的db_get命令是自由浮动的，没有锁，因为我不认为需要阅读需要。一个锁。
我看过这个代码一百万次，算法是气密的。所以我只是想知道我是否实现任何部分错误，使用锁错误等，或者如果有更好的方法来防止死锁（或甚至诊断死锁）与BerkeleyDB和草莓Perl？
UPDATE ：更具体地说，问题出现在Windows 2003服务器上（1.5 GB RAM，不确定是否重要）。我可以运行这个整个设置精细在我的Windows 7机器（4GB RAM）。我还开始使用以下内容打印锁定统计信息：
将此标志添加到环境创建中：
-MsgFile => $ dbFolder / lockData.txt
然后每60秒调用一次：
my $ status = $ env-> lock_stat_print（）;
print状态：$ status：\\\
;
状态始终返回为0，这是成功。这是最后一个统计报告：
29最后分配的储物柜ID
0x7fffffff当前最大未使用的储物柜ID
5锁定模式数量
1000可能的最大锁定数量
1000可能的最大锁定器数量
1000可能的最大锁定对象数量
40锁定对象分区数量
24当前锁的数量
42任一时刻的最大锁数
5任何一个桶中的最大锁数
0空分区被盗的最大数量
0任何一个分区被盗的最大锁数
29当前锁柜的数量
29任一时刻的最大锁柜数
6当前锁对象的数量
13最大数量锁定对象
1任何一个桶中的锁定对象的最大数量
0空分区被盗的最大数量
0任何一个分区被盗的最大数量
3121958请求的锁的总数
3121926已发布的锁的总数
0已升级的锁的总数
24降级的锁的总数
9310锁定请求不可用，由于冲突，我们等待了
0锁定请求由于冲突而不可用，我们没有等待
8死锁数量
1000000锁定超时值
0锁定数量已超时
1000000事务超时值
0超时的事务数
792KB锁定区域的大小
59需要等待的分区锁的数目（0％ ）
46等待任何分区锁的最大次数（0％）
0需要等待的对象队列操作数（0％）
27锁定器分配数必需等待（0％）
0需要等待的区域锁数量（0％）
1最大散列桶长度
其中我非常警惕：
8死锁数
这些死锁是如何发生的，它们是如何解决的？ （代码的所有部分仍在运行）。在这种情况下，什么是死锁？
解决方案总之，你需要做死锁检测。我可以看到两种可能性。首先，您可以使用
db_deadlock实用程序。第二，也许更方便，你可以在打开你的环境时指定-LockDetect标志，这个标志在 Perl docs forBerkeleyDB.pm。
在4.5.20版本中，这两种方法对我都有效。 （顺便说一下，您的版本）。
现在查看详细信息。
指定
-LockDetect标志是真的只是这样。有几个值可供选择。我选择了DB_LOCK_DEFAULT，它似乎工作正常。
运行
db_deadlock实用程序可以完成像这样：db_deadlock -h您的/ env / dir -v -t 3＃作为守护程序运行，每3秒检查一次
db_deadlock -h your / env / dir -v＃run once
db_deadlock手动：
此实用程序应作为后台守护程序运行，底层的Berkeley DB死锁检测接口应该以其他方式调用，每当有多个线程或进程访问数据库，并且至少有一个正在修改它。
我得出的结论是，两种方法都可以通过重复执行一个测试与两个作家和一个读者，这将死锁几次，同时快速连续数据库中的新条目（100每秒）或通过数据库中所有键的光标。
标志方法看起来很快处理死锁，他们在我的测试中没有变得明显。
另一方面，使用脚本在详细输出中运行
db_deadlock实用程序是有帮助的，查看它们如何阻止，然后在储物柜中止后，尤其是与db_stat实用程序：db_stat -Cl＃按锁定器分组的锁
db_stat -Co＃按对象分组的锁
db_stat -Cp＃need_dd = 1？
db_stat -CA＃以上所有加上
我缺乏专业知识来解释所有细节，但是你可以看到在阻塞的情况下有某些条目，而在其他的没有。另请参阅 Berkeley DB并发数据存储锁定约定一节 IWRITE
你问的是这些死锁是怎么发生的。无法完全说出，但我确实看到他们 发生并发访问。你也问他们是如何解决的。我不知道。在我的测试场景中，被阻止的脚本只会挂起。也许在你的场景中有人在你不知道死锁检测的情况下运行死锁检测？
为了完整性，你的应用程序可能只是挂起，因为线程尚未关闭资源退出前。可能会发生，如果你只是Ctrl-C一个进程，没有清理处理程序到位以关闭资源。但这似乎不是你的问题。
如果它确实成为你的问题，你应该查看在参考指南中处理数据存储和并发数据存储应用程序中的失败。
CDS和DS没有恢复的概念。由于CDS和DS不支持事务并且不维护恢复日志，因此它们无法运行恢复。如果数据库在DS或CDS中损坏，您只能删除它并重新创建它。 （取自 Berksley DB Book by Himanshu Yadava ）。
最后，在Oracle网站上有视频教程，包括由Margo Seltzer使用CDS 。
I've been having some problems with using BerkeleyDB. I have multiple instances of the same code pointed to a single repository of DB files, and everything runs fine for 5-32 hours, then suddenly there is a deadlock. The command prompts stop right before executing a db_get or db_put or cursor creation call. So I'm simply asking for the proper way to handle these calls. Here's my general layout:
This is how the environment and DBs are created:
my $env = new BerkeleyDB::Env ( -Home => "$dbFolder\\" , -Flags => DB_CREATE | DB_INIT_CDB | DB_INIT_MPOOL) or die "cannot open environment: $BerkeleyDB::Error\n"; my $unsortedHash = BerkeleyDB::Hash->new ( -Filename => "$dbFolder/Unsorted.db", -Flags => DB_CREATE, -Env => $env ) or die "couldn't create: $!, $BerkeleyDB::Error.\n";A single instance of this code runs, goes to a site and saves URLs to be parsed by another instance (I have the flag set so that every DB is locked when one is locked):
$lk = $unsortedHash->cds_lock(); while(@urlsToAdd){ my $currUrl = shift @urlsToAdd; $unsortedHash->db_put($currUrl, '0'); } $lk->cds_unlock();It periodically checks if a certain number of items are in Unsorted:
$refer = $unsortedHash->db_stat(); $elements = $refer->{'hash_ndata'};Before adding any element to any DB, it first checks all DBs to see if that element is already present:
if ($unsortedHash->db_get($search, $value) == 0){ $value = "1:$value"; }elsif ($badHash->db_get($search, $value) == 0){ $value = "2:$value"; ....This next code comes after, and many instances of it are run in parallel. First, it gets the next item in unsorted (that does not have the busy value '1'), then sets the value to busy '1', then does something with it, then moves the DB entry completely to another DB (it is removed from unsorted and stored in another DB):
my $pageUrl = ''; my $busy = '1'; my $curs; my $lk = $unsortedHash->cds_lock(); #lock, change status to 1, unlock ########## GET AN ELEMENT FROM THE UNSORTED HASH ####### while(1){ $busy = '1'; $curs = $unsortedHash->db_cursor(); while ($busy){ $curs->c_get($pageUrl, $busy, DB_NEXT); print "$pageUrl:$busy:\n"; if ($pageUrl eq ''){ $busy = 0; } } $curs->c_close(); $curs = undef; if ($pageUrl eq ''){ print "Database empty. Sleeping...\n"; $lk->cds_unlock(); sleep(30); $lk = $unsortedHash->cds_lock(); }else{ last; } } ####### MAKE THE ELEMENT 'BUSY' AND DOWNLOAD IT $unsortedHash->db_put($pageUrl, '1'); $lk->cds_unlock(); $lk = undef;And in every other place, if I call db_put or db_del on ANY DB, it is wrapped with a lock like so:
print "\n\nBad.\n\n"; $lk = $badHash->cds_lock(); $badHash->db_put($pageUrl, '0'); $unsortedHash->db_del($pageUrl); $lk->cds_unlock(); $lk = undef;However, my db_get commands are free-floating with no lock, because I don't think reading needs a lock.
I have looked over this code a million times and the algorithm is airtight. So I am just wondering if I am implementing any part of this wrong, using the locks wrong, etc. Or if there is a better way to prevent deadlocking (or even diagnose deadlocking) with BerkeleyDB and Strawberry Perl?
UPDATE: To be more specific, the problem is occurring on a Windows 2003 server (1.5 GB RAM, not sure if that is important). I can run this whole setup fine on my Windows 7 machine (4GB RAM). I also started printing out the lock stats using the following:
Adding this flag to the environment creation:
-MsgFile => "$dbFolder/lockData.txt"And then calling this every 60 seconds:
my $status = $env->lock_stat_print(); print "Status:$status:\n";The status is always returned as 0, which is success. Here is the last stat report:
29 Last allocated locker ID 0x7fffffff Current maximum unused locker ID 5 Number of lock modes 1000 Maximum number of locks possible 1000 Maximum number of lockers possible 1000 Maximum number of lock objects possible 40 Number of lock object partitions 24 Number of current locks 42 Maximum number of locks at any one time 5 Maximum number of locks in any one bucket 0 Maximum number of locks stolen by for an empty partition 0 Maximum number of locks stolen for any one partition 29 Number of current lockers 29 Maximum number of lockers at any one time 6 Number of current lock objects 13 Maximum number of lock objects at any one time 1 Maximum number of lock objects in any one bucket 0 Maximum number of objects stolen by for an empty partition 0 Maximum number of objects stolen for any one partition 3121958 Total number of locks requested 3121926 Total number of locks released 0 Total number of locks upgraded 24 Total number of locks downgraded 9310 Lock requests not available due to conflicts, for which we waited 0 Lock requests not available due to conflicts, for which we did not wait 8 Number of deadlocks 1000000 Lock timeout value 0 Number of locks that have timed out 1000000 Transaction timeout value 0 Number of transactions that have timed out 792KB The size of the lock region 59 The number of partition locks that required waiting (0%) 46 The maximum number of times any partition lock was waited for (0%) 0 The number of object queue operations that required waiting (0%) 27 The number of locker allocations that required waiting (0%) 0 The number of region locks that required waiting (0%) 1 Maximum hash bucket lengthOf which I am wary of this:
8 Number of deadlocksHow did these deadlocks occur, and how were they resolved? (all parts of the code are still running). What exactly is a deadlock, in this case?
解决方案In short, you need to do deadlock detection. I can see two possibilities to do that. First, you can use the
db_deadlockutility. Second, and perhaps more conveniently, you can specify the-LockDetectflag when opening your environment, a flag that's not exactly explained in depth in the Perl docs forBerkeleyDB.pm.Both ways appear to work fine for me in version 4.5.20. (What's your version, by the way?)
Now for the detail.
Specifying the
-LockDetectflag is really just that. There are a couple of values to choose from. I choseDB_LOCK_DEFAULTand it appeared to work just fine. With more clues as to what's going on you could certainly get more fancy.Running the
db_deadlockutility could be done like this:db_deadlock -h your/env/dir -v -t 3 # run as daemon, check every 3 seconds db_deadlock -h your/env/dir -v # run onceHere's a quote from the
db_deadlockmanual:This utility should be run as a background daemon, or the underlying Berkeley DB deadlock detection interfaces should be called in some other way, whenever there are multiple threads or processes accessing a database and at least one of them is modifying it.
I arrived at the conclusion that both ways do work fine by repeatedly performing a test with two writers and one reader, which would deadlock a couple times while putting new entries in the database in rapid succession (100 per second), or going through a cursor of all keys in the database.
The flag method appears to deal with deadlocks very quickly, they didn't become noticeable in my tests.
On the other hand, running the
db_deadlockutility with verbose output in paralles with the scripts is instructive in that you see how they block and then continue after lockers have been aborted, especially when combined with thedb_statutility:db_stat -Cl # Locks grouped by lockers db_stat -Co # Locks grouped by object db_stat -Cp # need_dd = 1 ? db_stat -CA # all of the above plus moreI lack the expertise to explain all the details, but you can see that in blocked situations there are certain entries there while in others there aren't. Also see the section entitled Berkeley DB Concurrent Data Store locking conventions(what is
IWRITE?) in the Berkeley DB Programmer's Reference Guide.You're asking how these deadlocks did occur. Can't say exactly, but I do see that they are occurring with concurrent access. You're also asking how they were resolved. I have no idea. In my test scenarios, blocked scripts will simply hang. Maybe in your scenario someone ran deadlock detection without you knowing about it?
For completeness, your application might simply hang because a thread has not closed resources before exiting. Might happen if you just Ctrl-C a process and there is no clean-up handler in place to close resources. But that doesn't appear to be your problem.
If it does become your problem, you should review the section on Handling failure in Data Store and Concurrent Data Store applications in the Reference Guide.
CDS and DS have no concept of recovery. Since CDS and DS don't support transactions and don't maintain a recovery log, they cannot run recovery. If the database gets corrupted in DS or CDS, you can only remove it and recreate it. (Taken moreless verbatim from the Berkeley DB Book by Himanshu Yadava.)
Finally, there are video tutorials on the Oracle site, including one on using CDS by Margo Seltzer.
这篇关于使用Perl访问BerkeleyDB的正确方法是什么？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
