无法从数据库中清除600,000个删除存根 [英] Unable get 600,000 deletion stubs ro purge from database

问题描述

更新:我已经发布了:show数据库mycleandb.nsf，我看到仍然有835,000个删除存根.我似乎无法删除它们.

Update: I have issued a: show database mycleandb.nsf and I can see there are still 835,000 deletion stubs. I can't seem to remove them.

摘要: 我们有一个大型数据库，现在可以开始准备将其缩减为更易于管理的大小. 我删除了文档，但是剩下一个数据库，它似乎无法复制或复制，出现错误:无法扩展ID表-内存不足.

Summary: We have a large database that we are now ready to start trimming down to a more manageable size. I deleted the documents, but I am left with a database it appears I can't copy or replicate, receiving error: Unable to extend an ID table - insufficient memory.

详细信息: 当前的实时数据库中大约有1,400,000个文档，我们可以立即将其减少到650,000.通过进一步的设计\建筑工作，我们可以将其减少到300,000.该Web应用程序群集在5台服务器上，其中1台是应用程序集线器"，反向代理后面有3台HTTP \ SSO服务器，而第5台服务器是外部服务的访问点. HUB每30分钟启动一次PUSH \ PULL复制(无文档限制)，日志显示无复制错误.事务日志记录是针对服务器上的运行时性能设置的，备份是在不在群集中的另一台后台服务器上执行的.我们每天进行存档，每天存档约1,500个文档.

Detail: Current live database has roughly 1,400,000 docs in it, we can immediately reduce it to 650,000. With some further design\ architectural work we can reduce this to 300,000. This web application is clustered on 5 servers, 1 being the application "hub", with 3 HTTP\SSO servers behind a reverse proxy and the 5th server is the access point for external services. The HUB initiates a PUSH\PULL replication (with no document restrictions) every 30mins, logs show no replication errors. Transaction logging is set for Run Time performance on servers, back up is performed on another back server not in cluster. We run daily archiving that archives roughly 1,500 docs a day.

到目前为止，将近2年的时间，应用程序运行良好...除了这里和那里的一些噩梦:-).但是，我们终于可以精简数据库了.

So far, nearly 2 years on, application has run fine...apart from a few nightmares here and there :-). But, we are finally at point we can trim the databases down.

在进行任何工作之前，OS都会备份实时数据库，并将其复制到测试服务器.在测试服务器上，我执行了以下操作:CL复制mylivedb.nsf mycleandb.nsf 然后，在mycleandb.nsf中，我删除了文档以将数据库减少到650,000个文档.

Prior to any work I OS backed up the live DB, and copied it to a test server. On a test server I did a: CL copy mylivedb.nsf mycleandb.nsf In mycleandb.nsf I then deleted docs to reduce the DB to 650,000 documents.

我所需要的只是一个新副本，而不是副本，请参见下面的副本问题，因此我发出了另一个命令...给我一个没有删除存根的新副本，据我了解，COPY不会复制del的存根: CL复制mycleandb.nsf mycleandbNEW.nsf ...这还可以确保如果网络上有任何流氓副本，我们也不会将其文档全部归还(并非我们所有的服务器都是8.5.3，因此我们无法使用数据库属性来设置剪切删除存根的截止日期)

All I'm after is a new copy, not a replica, see replica issue below, so I issued another command...to give me a new copy without deletion stubs, it's my understanding that a COPY does not copy del' stubs.: CL copy mycleandb.nsf mycleandbNEW.nsf ...this also ensures that if any rogue replica is out on the network, we don't get allt he docs back in (not all our servers are 8.5.3, so we can't use the Database Property to set a cut off date for deletion stubs)

但是我得到一个:无法扩展ID表-内存不足. 试过这个: CL复制mycleandb.nsf mycleandbNEW.nsf副本 ...是同一件事.

But I got a: Unable to extend an ID table - insufficient memory. Tried this: CL copy mycleandb.nsf mycleandbNEW.nsf REPLICA ...same thing.

尝试在此处找到我的mycleandb.nsf的步骤: http://www-01.ibm.com/support/docview.wss ?uid = swg21220384 ...是同一件事.

Tried steps found here to my mycleandb.nsf: http://www-01.ibm.com/support/docview.wss?uid=swg21220384 ...same thing.

按此处所述将清除间隔更改为0天，并将清除日期设置为将来: http://www-01.ibm.com/support/docview.wss ?uid = swg21095683 ...是同一件事.

Changed purge interval to 0 days as described here, also set purge date into future: http://www-01.ibm.com/support/docview.wss?uid=swg21095683 ...same thing.

然后运行: 加载紧凑的mycleandb.nsf -B ...是同一件事.

Then ran a: load compact mycleandb.nsf -B ...same thing.

我在这里看到了这个线程，这是类似的情况，除了，我们在clustering \复制上没有任何问题...但是！ http://www-10.lotus.com/ldd/nd85forum .nsf/DateAllFlatWeb/74d3e0f5467f843885257aaf0081abe5?OpenDocument

I have seen this thread here, which is a similar situation, except, we don't have any issues with clustering\ replication...yet! http://www-10.lotus.com/ldd/nd85forum.nsf/DateAllFlatWeb/74d3e0f5467f843885257aaf0081abe5?OpenDocument

因此，我有一个数据库，我已从其中删除了文档，可以打开它，它似乎可以运行，但是我不能复制\复制它.

So, I have a DB I have deleted docs from, I can open it, it appears to function, but I can't copy\ replicate it.

复制\群集问题: 我以前见过此错误:无法扩展ID表-内存不足...而我所做的是删除我们的一台主服务器，操作系统将数据库复制到有问题的服务器上，然后重新启动.

Replication\ Cluster Issue: I have seen this error before: Unable to extend an ID table - insufficient memory...and what I have done, is drop one of our primary servers, OS copy the DB to server with issue, restart.

我一直使用Admin Client \ Dom控制台创建新副本，但是在这种情况下，每当执行此操作时，它就会用文档填充数据库，但是清除间隔在所有服务器上都是正确的.新副本是从HUB服务器创建的，该服务器每30分钟对所有集群伙伴启动PUSH \ PULL复制，因此，如果这是一个删除存根问题，它会一直显示出来吗?

I have always created new replicas using Admin Client\ Dom console, but in this case, whenever I do it floods the database with documents, but purge intervals are correct on all servers. The new replicas have been created from the HUB server, which initiates PUSH\PULL replication with all the cluster mates every 30mins, so if this was a deletion stub issue, it would manifest itself all the time?

我将在明天的NotesPeek中查看是否有任何删除存根，但是我不确定如何进行.

I will NotesPeek it tomorrow to see if any deletion stubs are still there, but I am not sure how to proceed.

感谢收到任何评论或建议.

Any comments or suggestions gratefully received.

推荐答案

根据我的经验，当修改的文档+存根的总数太高时，Domino中实际上清除删除存根的代码无法正常工作.我认为，这可能是Notes API内部结构中的一个很深的局限性，它与必须使用的算法结合使用-要查看在清除间隔日期之前修改的所有笔记-可能非常大.

In my experience, the code in Domino that actually purges deletion stubs cannot work when the total number of modified docs + stubs is too high. I think it's probably a limitation that's pretty deep in the Notes API internals, combined with the algorithm that has to be used -- which is to look at all notes modified before the purge interval date -- a potentially very large number.

您最好的选择是制作本地非副本并重新部署.

Your best option may be to make a local non-replica copy and re-deploy.

我前一段时间对此进行了研究.我找到了一些Lotus Notes代码，这些代码使用Notes C API调用清除了张贴在各个博客上的删除存根. (我认为原始代码可能来自Rod Whitely，但我不确定.)下面是我的版本代码.

I researched this a while back. I found some LotusScript code that uses Notes C API calls to purge deletion stubs posted on various blogs. (I think the original code may have come from Rod Whitely, but I'm not sure.) My version code is below.

在文档+存根总数介于2到3百万之间的数据库上运行时，它得到无法扩展ID表-内存不足".我从来没有联系过IBM支持部门，因为这对我来说只是一个附带项目.我最终只是制作生产数据库的非副本副本，然后将清除间隔设置得足够低，以使存根的数量再也不会变得太高.

When run on a database where the total of docs + stubs is somewhere between 2 and 3 million, it gets the "Unable to extend an ID table - insufficient memory". I never did contact IBM support about this, as it was really just a side project for me. I ended up just making non-replica copies of the production databases and then setting the purge interval low enough so that the number of stubs never got too high again.

Declare Private Sub IDDestroyTable Lib wAPIModule Alias "IDDestroyTable" _
( ByVal hT As Long)
Declare Private Function IDScan Lib wAPIModule Alias "IDScan" _
( ByVal hT As Long, ByVal F As Integer, ID As Long) As Integer
Declare Private Function NSFDbOpen Lib wAPIModule Alias "NSFDbOpen" _
( ByVal P As String, hDB As Long) As Integer
Declare Private Function NSFDbClose Lib wAPIModule Alias "NSFDbClose" _
( ByVal hDB As Long) As Integer
Declare Private Function NSFDbGetModifiedNoteTable Lib wAPIModule Alias "NSFDbGetModifiedNoteTable" _
( ByVal hDB As Long, ByVal C As Integer, ByVal S As Currency, U As Currency, hT As Long) As Integer
Declare Private Function NSFNoteDelete Lib wAPIModule Alias "NSFNoteDelete" _
( ByVal hDB As Long, ByVal N As Long, ByVal F As Integer) As Integer
Declare Private Function OSPathNetConstruct Lib wAPIModule Alias "OSPathNetConstruct" _
( ByVal NullPort As Long, ByVal Server As String, ByVal FIle As String, ByVal PathNet As String) As Integer
Declare Private Sub TimeConstant Lib wAPIModule Alias "TimeConstant" _
( ByVal C As Integer, T As Currency)

Function countAndDeleteStubsByOpenDatabase(db As NotesDatabase, deleteFlag As boolean) As Long

        If db Is Nothing GoTo bail

        Dim forever As Currency
        Dim last As Currency
        Dim hT As Long
        Dim RRV As Long
        Dim hDB As Long
        Dim path As String
        Dim nStubs As Long
        Dim ret As integer

        On Error GoTo bail  

        path = Space(1024)
        Call OSPathNetConstruct(0, db.Server, db.FilePath, path)
        Call NSFDbOpen(path, hDB)

        Call TimeConstant(2, forever)
        Call NSFDbGetModifiedNoteTable(hDB, &H7FFF, forever, last, hT)

        nStubs = 0

        ret = IDScan(hT, True, RRV)

        While Not (ret = 0)
            If RRV < 0 Then
                If (deleteFlag = true) Then 
                    NSFNoteDelete hDB, RRV And &H7FFFFFFF, &H0201 
                End If
                nStubs = nStubs + 1
            End If
            ret = IDScan(hT, False, RRV)
        Wend

        IDDestroyTable hT
        NSFDbClose hDB

        If deleteFlag = True Then
            Print "Deleted " + CStr(nStubs) + " stubs for " + db.Filepath  + " on " + db.Server
        Else
            Print "Counted " + CStr(nStubs) + " stubs for " + db.Filepath  + " on " + db.Server 
        End If


        countAndDeleteStubsByOpenDatabase = nStubs

        On Error GoTo 0

        Exit Function

        bail:
            Print "Unexpected error in countAndDeleteStubsByOpenDatabase. Line " + CStr(Erl) + " " + CStr(Err()) + " " + Error(Err())
            countAndDeleteStubsByOpenDatabase = 0

    End Function

我从没有尝试过，但是我也想到可以修改此代码以仅搜索最近一天的存根，将其删除，然后再返回一天并获取最新的存根存根，删除这些存根等.但是，由于您必须将C TIMEDATE结构作为货币"字段处理，因此在LotusScript中可能并非那么容易.您可能必须在C中执行此操作.当然，首先删除最新的存根与实际要执行的操作完全相反，但是此策略可能会解决对idtables的限制.

I never tried it, but it also occurred to me that it might be possible to modify this code to just search for the most recent one day's worth of stubs, delete them, then go back one more day and get the most recent stubs, delete those, etc. It might not be all that easy to do in LotusScript, though, given that you have to deal with the C TIMEDATE struc as a Currency field. You might have to do it in C. Of course, deleting the newest stubs first is the exact reverse of what you want to do in practice, but this strategy might work around the limitation on idtables.

这篇关于无法从数据库中清除600,000个删除存根的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！