如何确保Solr/Lucene不会因java.lang.OutOfMemoryError而死? [英] How to make sure Solr/Lucene won't die with java.lang.OutOfMemoryError?

问题描述

我真的感到困惑，为什么即使索引只有几GB的内存，在索引过程中它仍会死于java.lang.OutOfMemoryError.

I'm really puzzled why it keeps dying with java.lang.OutOfMemoryError during indexing even though it has a few GBs of memory.

是否存在根本原因，为什么它需要手动调整配置文件/jvm参数，而不是仅仅弄清楚有多少可用内存并将其自身限制于此?除了Solr之外，没有其他程序会遇到这种问题.

Is there a fundamental reason why it needs manual tweaking of config files / jvm parameters instead of it just figuring out how much memory is available and limiting itself to that? No other programs except Solr ever have this kind of problem.

是的，每次发生此类崩溃时，我都可以不断调整JVM堆大小，但这一切都是倒退的.

Yes, I can keep tweaking JVM heap size every time such crashes happen, but this is all so backwards.

以下是有关此类最新崩溃的堆栈跟踪:

Here's stack trace of the latest such crash in case it is relevant:

SEVERE: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Arrays.java:3209)
    at java.lang.String.<init>(String.java:216)
    at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122)
    at org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:169)
    at org.apache.lucene.search.FieldCacheImpl$StringIndexCache.createValue(FieldCacheImpl.java:701)
    at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:208)
    at org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:676)
    at org.apache.lucene.search.FieldComparator$StringOrdValComparator.setNextReader(FieldComparator.java:667)
    at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:94)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:245)
    at org.apache.lucene.search.Searcher.search(Searcher.java:171)
    at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988)
    at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884)
    at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
    at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
    at java.lang.Thread.run(Thread.java:619)

推荐答案

查看堆栈跟踪，看起来您正在执行搜索并按字段排序.如果需要按字段排序，则Lucene内部需要将字段中所有术语的所有值加载到内存中.如果该字段包含大量数据，则很有可能您的内存不足.

Looking at the stack trace, it looks like you are performing a search, and sorting by a field. If you need to sort by a field, internally Lucene needs to load up all the values of all the terms in the field into memory. If the field contains a lot of data, then it is very possible that you may run out of memory.

这篇关于如何确保Solr/Lucene不会因java.lang.OutOfMemoryError而死?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！