新IndexWriter()上的LockObtainFailedException [英] LockObtainFailedException at new IndexWriter()

问题描述

我正在使用Crawler控制器对中型网站的所有页面进行爬网.它会随机抓取2-3页，然后在IndexWriter上造成锁定

I am using Crawler controller to crawl all pages of a medium website. It randomly crawls 2-3 pages and then it causes a lock on the IndexWriter

Directory dir = FSDirectory.open(new File(index));
IndexWriterConfig conf = new IndexWriterConfig(org.apache.lucene.util.Version.LUCENE_41,new StandardAnalyzer(org.apache.lucene.util.Version.LUCENE_41));
writer = new IndexWriter(dir, conf);  // line which throws lock exception. 

日志:

来自:SiteSearch.KCCrawlerController.(80):锁定获取超时:NativeFSLock @ D:\ Websites \ ccc \ WEB-INF \ lucene-index \ en \ write.lock:2014年5月8日10:57:55org.apache.lucene.store.LockObtainFailedException:锁定获取超时:NativeFSLock @ D:\ Websites \ ccc \ WEB-INF \ lucene-index \ en \ write.lock在org.apache.lucene.store.Lock.obtain(Lock.java:84)在org.apache.lucene.index.IndexWriter.(IndexWriter.java:636)在SiteSearch.KCCrawlerController.(KCCrawlerController.java:80)在org.apache.jsp.monitors.siteSearchIndexer_jsp._jspService(siteSearchIndexer_jsp.java:66)在org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)在javax.servlet.http.HttpServlet.service(HttpServlet.java:717)在org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:386)在org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)在org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)在javax.servlet.http.HttpServlet.service(HttpServlet.java:717)在org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)在org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)在com.tridion.ambientdata.web.AmbientDataServletFilter.doFilter(AmbientDataServletFilter.java:255)在org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)在org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)在adminV3.ugc.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)在org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)在org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)在org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)在org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)在org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)在org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)在org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)在org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)在org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:429)在org.apache.coyote.ajp.AjpAprProtocol $ AjpConnectionHandler.process(AjpAprProtocol.java:384)在org.apache.tomcat.util.net.AprEndpoint $ Worker.run(AprEndpoint.java:1665)在java.lang.Thread.run(未知来源)

From: SiteSearch.KCCrawlerController.(80): Lock obtain timed out: NativeFSLock@D:\Websites\ccc\WEB-INF\lucene-index\en\write.lock: 05/08/2014 10:57:55 org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@D:\Websites\ccc\WEB-INF\lucene-index\en\write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:84) at org.apache.lucene.index.IndexWriter.(IndexWriter.java:636) at SiteSearch.KCCrawlerController.(KCCrawlerController.java:80) at org.apache.jsp.monitors.siteSearchIndexer_jsp._jspService(siteSearchIndexer_jsp.java:66) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:386) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.tridion.ambientdata.web.AmbientDataServletFilter.doFilter(AmbientDataServletFilter.java:255) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at adminV3.ugc.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:429) at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:384) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665) at java.lang.Thread.run(Unknown Source)

添加jsp: http://example.com/en/consulting/diagnostics.jsp?crawler=yes
来自:SiteSearch.KCCrawler.visit(95):流关闭:2014/05/0810:57:55 java.io.IOException:流在以下时间关闭org.apache.jasper.runtime.JspWriterImpl.ensureOpen(JspWriterImpl.java:204)在org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:312)在org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:342)在SiteSearch.KCCrawler.visit(KCCrawler.java:95)处edu.uci.ics.crawler4j.crawler.WebCrawler.processPage(WebCrawler.java:306)在edu.uci.ics.crawler4j.crawler.WebCrawler.run(WebCrawler.java:189)在java.lang.Thread.run(未知来源)

Adding jsp: http://example.com/en/consulting/diagnostics.jsp?crawler=yes
From: SiteSearch.KCCrawler.visit(95): Stream closed: 05/08/2014 10:57:55 java.io.IOException: Stream closed at org.apache.jasper.runtime.JspWriterImpl.ensureOpen(JspWriterImpl.java:204) at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:312) at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:342) at SiteSearch.KCCrawler.visit(KCCrawler.java:95) at edu.uci.ics.crawler4j.crawler.WebCrawler.processPage(WebCrawler.java:306) at edu.uci.ics.crawler4j.crawler.WebCrawler.run(WebCrawler.java:189) at java.lang.Thread.run(Unknown Source)

为什么会出现此异常?任何帮助.

Why am I getting this exception? Any help.

当我第一次运行Indexer时，它会成功完成，并抛出以下异常.如果我对此进行搜索，那么我将成功获得结果.但是，如果我再次运行Indexer，它将引发上述锁异常.它还显示了我的控制器类被调用了两次.

When I run the Indexer first time, it completes successfully with the below exception thrown. If I run the search on this, I get my results successfully. However if I run the Indexer again, it throws the lock exception mentioned above. It also shows that my controller class is called twice.

org.apache.catalina.core.StandardWrapperValve调用SEVERE:Servlet jsp的Servlet.service()抛出异常java.io.IOException:流关闭于org.apache.jasper.runtime.JspWriterImpl.ensureOpen(JspWriterImpl.java:204)在org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:115)在org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:188)在org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:118)在org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:77)在org.apache.jsp.monitors.siteSearchIndexer_jsp._jspService(siteSearchIndexer_jsp.java:82)在org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)在javax.servlet.http.HttpServlet.service(HttpServlet.java:717)处org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:386)

org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet jsp threw exception java.io.IOException: Stream closed at org.apache.jasper.runtime.JspWriterImpl.ensureOpen(JspWriterImpl.java:204) at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:115) at org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:188) at org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:118) at org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:77) at org.apache.jsp.monitors.siteSearchIndexer_jsp._jspService(siteSearchIndexer_jsp.java:82) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:386)

org.apache.jasper.servlet.JspServlet.serviceJspFile中的

at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)

推荐答案

引用Javadocs-

Quoting the Javadocs -

打开IndexWriter会为使用中的目录创建一个锁定文件.尝试在同一目录中打开另一个IndexWriter会导致LockObtainFailedException.如果同一目录上的IndexReader用于从中删除文档，也会引发LockObtainFailedException.索引."

"Opening an IndexWriter creates a lock file for the directory in use. Trying to open another IndexWriter on the same directory will lead to a LockObtainFailedException. The LockObtainFailedException is also thrown if an IndexReader on the same directory is used to delete documents from the index."

"IndexWriter实例完全是线程安全的，这意味着多个线程可以同时调用其任何方法.如果您的应用程序需要外部同步，则不应在IndexWriter实例上进行同步，因为这可能会导致死锁；请使用您自己的(非-Lucene)对象."

"IndexWriter instances are completely thread safe, meaning multiple threads can call any of its methods, concurrently. If your application requires external synchronization, you should not synchronize on the IndexWriter instance as this may cause deadlock; use your own (non-Lucene) objects instead."

https://lucene.apache.org/core/4_1_0/core/org/apache/lucene/index/IndexWriter.html

您是否正在为要爬网的每个页面创建IndexWriter的新实例?

Are you creating new instances of IndexWriter for each page that you are crawling?

这篇关于新IndexWriter()上的LockObtainFailedException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！