Solr在Tomcat上，Windows OS占用所有内存 [英] Solr on Tomcat, Windows OS consumes all memory

问题描述

更新

我已经配置了xms(初始内存)和xmx(最大内存分配jvm参数)，在重新启动后，我连接了Visual VM以监视Tomcat的内存使用情况. Tomcat看起来还不错，内存消耗在已定义的jvm参数范围内. 因此，似乎文件系统缓冲区正在消耗所有剩余的内存，并且不会丢失内存?有没有办法处理这种行为，例如更改nGram大小或directoryFactory?

I've configured both xms (initial memory) and xmx (maximum memory allocation jvm paramters, after a restart i've hooked up Visual VM to monitor the Tomcat memory usage. While the indexing process is running, the memory usage of Tomcat seems ok, memory consumption is in range of defined jvm params. (see image)
So it seems that filesystem buffers are consuming all the leftover memory, and does not drop memory? Is there a way handle this behaviour, like change nGram size or directoryFactory?

我对Solr和Tomcat还是陌生的，但是我们开始:

I'm pretty new to Solr and Tomcat, but here we go:

OS Windows Server 2008

OS Windows server 2008

• 4 Cpu
• 8 GB Ram

Tomcat Service版本7.0(64位)

Tomcat Service version 7.0 (64 bit)

• 仅运行Solr
• 未设置可选的JVM参数，但通过GUI进行Solr配置

Solr版本4.5.0.

Solr version 4.5.0.

• 一个Core实例(用于查询和索引编制)

架构配置:

• minGramSize ="2" maxGramSize ="20"
• 大多数字段都存储为="true"(必填)

Solr配置:

• ramBufferSizeMB:100
• maxIndexingThreads:8
• directoryFactory:MMapDirectory
• 自动提交:maxdocs 10000，maxtime 15000，opensearcher false
• 缓存(默认值):
  filtercache初始大小:512大小:512自动预热:0
  queryresultcache初始大小:512大小:512自动预热:0
  documentcache initialsize:512 size:512 autowarm:0
  

• ramBufferSizeMB: 100
• maxIndexingThreads: 8
• directoryFactory: MMapDirectory
• autocommit: maxdocs 10000, maxtime 15000, opensearcher false
• cache (defaults):
  filtercache initialsize:512 size: 512 autowarm: 0
  queryresultcache initialsize:512 size: 512 autowarm: 0
  documentcache initialsize:512 size: 512 autowarm: 0
  

我们正在使用.Net服务(基于Solr.Net)在单个Solr Core实例上更新和插入文档.发送到Solr的文档大小从1 Kb到8Mb 不等，我们正在使用一个或多个线程批量发送文档. Solr索引的当前大小约为15GB.

We're using a .Net Service (based on Solr.Net) for updating and inserting documents on a single Solr Core instance. The size of documents sent to Solr vary from 1 Kb up to 8Mb, we're sending the documents in batches, using one or multiple threads. The current size of the Solr Index is about 15GB.

索引服务大约需要3到4个小时运行一次，以完成对Solr的所有插入和更新.在建立索引过程的过程中， Tomcat进程的内存使用情况 会不断增长到> 7GB Ram ，即使在24小时之后也不会减少. 重新启动Tomcat或Solr Admin中的Reload Core之后，内存将降回1 a 2 GB Ram.内存泄漏?

The indexing service is running around 3 a 4 hours to complete all inserts and updates to Solr. While the indexing process is running the Tomcat process memory usage keeps growing up to > 7GB Ram and does not reduce, even after 24 hours. After a restart of Tomcat, or a Reload Core in the Solr Admin the memory drops back to 1 a 2 GB Ram. Memory leak?

是否可以为Tomcat上的Solr进程配置最大内存使用量? 还有其他选择吗?最佳做法?

Is it possible to configure the max memory usage for the Solr process on Tomcat? Are there other alternatives? Best practices?

谢谢

推荐答案

我有以下设置(尽管问题小得多)... 5000个文档，文档大小从1MB到30MB. 对于2 CPU/2GB系统上的Tomcat进程，我们要求在1GB以下运行

I have the following setup (albeit a much smaller problem)... 5000 documents, document sizes range from 1MB to 30MB. We have a requirement to run under 1GB for the Tomcat process on a 2 CPU / 2GB system

经过一番实验后，我想到了JAVA的这些设置.

After bit of experimentation I came up with these settings for JAVA.

-Xms448m
-Xmx768m
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:ParallelCMSThreads=4
-XX:PermSize=64m
-XX:MaxPermSize=64m
-XX:NewSize=384m
-XX:MaxNewSize=384m
-XX:TargetSurvivorRatio=90
-XX:SurvivorRatio=6
-XX:+CMSParallelRemarkEnabled
-XX:CMSInitiatingOccupancyFraction=55
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+OptimizeStringConcat
-XX:+UseCompressedOops
-XX:MinHeapFreeRatio=5
-XX:MaxHeapFreeRatio=5

这些有帮助，但是即使有这么小的数据集，我也遇到了OutOfMemory和Tomcat使用过多内存的问题.

These helped but I encountered issues with the OutOfMemory and Tomcat using too much memory even with such a small dataset.

到目前为止，我确定的解决方案或事物/配置看起来很合适，如下所示:

Solution Or things/configuration I have set so far that seem to hold well are as follows:

1. 禁用除QueryResultCache之外的所有缓存
2. 查询中不包括文本/内容字段，仅包含ID
3. 请勿使用大于10的行，并且不包括突出显示.
4. 如果您正在使用突出显示(这是最大的罪魁祸首)，请先从查询中获取文档标识符，然后再次使用突出显示和包含ID字段的搜索词进行查询.

最后是内存问题.我不得不勉强地实现一种非正统的方法来解决tomcat/java内存占用问题(因为Java从未将内存返还给操作系统).

Finally for the memory problem. I had to grudgingly implement an unorthodox approach to solve the tomcat/java memory hogging issue (as java never gives back memory to the OS).

我创建了一个内存调试器服务，该服务以调试特权运行，并调用Windows API来强制tomcat进程释放内存.我还拥有一个全局互斥锁，以防止在通话时发生这种情况时访问tomcat.

I created a memory governor service that runs with debug privilege and calls windows API to force tomcat process to release memory. I also have a global mutex to prevent access to tomcat while this happens when a call comes in.

令人惊讶的是，如果您没有选择权来控制对Tomcat的访问，则这种方法行之有效，但并非没有风险.

Surprisingly this approach is working out well but not without its own perils if you do not have the option to control access to Tomcat.

如果您找到更好的解决方案/配置更改，请告诉我们.

If you find a better solution/configuration changes please let us know.

这篇关于Solr在Tomcat上，Windows OS占用所有内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！