Solr-Java内存不足 [英] Solr - Java out of memory

问题描述

我有一个可创建相当大的Solr 3.6索引的应用程序. 300GB，每天有1B个文件，分为10个核心.索引的效果很好，我正在使用循环算法在核心之间平均分配文档.搜索对我也非常有用，直到返回结果集大于10万个文档.

I have an application that creates a rather large Solr 3.6 index, approx. 300GB with 1B documents divided into 10 cores each day. Indexing works great, and I’m using a round-robin algorithm to distribute the docs evenly between the cores. Searches work great for me too up to the point that the return result set is greater than 100K+ documents.

到那时，我收到一个Java错误:OutOfMemoryError或SolrException:解析错误

At that point, I get a java error returned: either OutOfMemoryError or SolrException: parsing error

我的搜索很简单，没有使用通配符，排序或多面搜索，但似乎在返回结果之前先缓冲整个结果集.服务器上的物理内存为256G，并且正在运行Solaris10.我正在使用32位的默认Java，但也尝试了32位和64位的Java 7.

My searches are simple, not using wildcards or sorting or faceted search, yet it seems to buffer the entire result set before returning it. The physical memory on my server is 256G and I am running Solaris 10. I’m using the default java in 32 bit, but have also tried java 7 in 32 and 64 bit.

当我使用64位Java时，我可以使用–Xmx选项增加足够的最大内存以返回1M +文档，但是实际上它仅需要一个Solr进程就拥有我拥有的所有内存.

When I use 64 bit java, I am able to increase the max memory enough to return 1M+ documents with the –Xmx option, but it requires practically all the memory I have for just a single Solr process.

除了用数百个小索引重新设计我的应用程序之外，还有人对如何在不使用大量RAM的情况下从Solr中获取大型搜索结果集有任何建议吗?

Other than re-designing my application with hundreds of tiny indexes, does anyone have any suggestions on how to get large search result sets out of Solr without huge amounts of RAM?

推荐答案

您可以尝试禁用各种缓存(例如filterCache，queryResultCache和documentCache).这可能会损害性能，但可能会提供一些呼吸空间.

You can try disabling various caches (such as filterCache, queryResultCache, and documentCache). This will likely hurt the performance but might give some breathing space.

如果Solr HTTP/XML响应很大，则可以考虑将Solr放在同一JVM下甚至使用原始Lucene来节省XML开销.

If your Solr HTTP/XML responses are big, you can consider placing Solr under the same JVM or even using raw Lucene to save on XML overhead.

除此之外，恐怕您还需要研究分片.

Other than that I'm afraid you will need to look into sharding.

这篇关于Solr-Java内存不足的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！