MongoDB将文档预加载到RAM中以提高性能 [英] MongoDB preload documents into RAM for better performance
问题描述
我希望MongoDB将查询结果保存在RAM中更长的时间(如果有可用内存，则说30分钟).是否有可能?或者，有什么方法可以确保在后续查询之前将数据预先加载到RAM中.
I want MongoDB to hold query results in RAM for longer period of time (say 30 minutes if memory is available). Is it possible? OR is there any way i can make sure that the data is pre-loaded into RAM before subsequent queries on it.
实际上，我想知道MongoDB的简单查询结果性能.我有一个具有10GB RAM的专用服务器，我的db.stats()如下；
In fact i am wondering about simple query results performance by MongoDB. I have a dedicated server with 10GB RAM and my db.stats() are as follows;
db.stats();
{
"db": "test",
"collections":16,
"objects":625690,
"avgObjSize":68.90,
"dataSize":43061996,
"storageSize":1121402888,
"numExtents":74,
"indexes":25,
"indexSize":28207200,
"fileSize":469762048,
"nsSizeMB":16,
"ok":1
}
Now when i query single document (as mentioned here) from a web service it loads in 1.3 seconds. Subsequent calls of same queries gives response in 400ms and then after few seconds, it again starts taking 1.3 seconds. Looks like MongoDB has lost the previous queried document from Memory, where as there is no other queries asking for data mapped to RAM.
请对此进行解释，并让我知道以任何方式使后续查询更快地响应.
Please explain this and let me know any way to make subsequent queries faster responding.
推荐答案
您在初始查询中观察到的性能问题很可能是以下问题之一(按大概的顺序):
Your observed performance problem on an initial query is likely one of the following issues (in rough order of likelihood):
1)您的应用程序/Web服务有一些开销，需要在首次请求时进行初始化(即分配内存，设置连接池，解析DNS等).
1) Your application / web service has some overhead to initialize on first request (i.e. allocating memory, setting up connection pools, resolving DNS, ...).
2)您所请求的索引或数据尚未存储在内存中，因此需要加载.
2) Indexes or data you have requested are not yet in memory, so need to be loaded.
3)查询优化器可能要花更长的时间才能处理第一个请求，因为它正在比较您的查询模式的计划执行情况.
3) The Query Optimizer may take a bit longer to run on the first request, as it is comparing the plan execution for your query pattern.
通过mongo shell测试查询，并确定开销是与MongoDB还是与Web服务相关(而不是像您那样同时对两者进行计时)，将非常有帮助.
It would be very helpful to test the query via the mongo shell, and isolate whether the overhead is related to MongoDB or your web service (rather than timing both, as you have done).
以下是与MongoDB相关的一些注释.
Following are some notes related to MongoDB.
MongoDB在内存中没有缓存"时间.它使用内存映射文件作为磁盘I/O，并且内存中的文档基于您的活动查询(您最近加载的文档/索引)以及可用内存.操作系统的虚拟内存管理器负责缓存，通常会遵循最近最少使用(LRU)确定要换出内存的页面的算法.
MongoDB doesn't have a "caching" time for documents in memory. It uses memory-mapped files for disk I/O and the documents in memory are based on your active queries (documents/indexes you've recently loaded) as well as the available memory. The operating system's virtual memory manager is in charge of caching, and typically will follow a Least-Recently Used (LRU) algorithm to decide which pages to swap out of memory.
预期的行为是，随着时间的流逝，MongoDB将增长为使用所有可用内存来存储您的活动工作数据集.
The expected behaviour is that over time MongoDB will grow to use all free memory to store your active working data set.
查看您提供的db.stats()编号(并假设这是您的 only 数据库)，看来您的数据库大小当前约为1Gb，因此您应该能够将所有数据保持在10Gb之内RAM，除非:
Looking at your provided db.stats() numbers (and assuming that is your only database), it looks like your database size is current about 1Gb so you should be able to keep everything within your 10Gb total RAM unless:
- 还有其他进程争夺内存
- 您已重新启动
mongod服务器，但尚未请求这些文档/索引
- there are other processes competing for memory
- you have restarted your
mongodserver and those documents/indexes haven't been requested yet
在MongoDB 2.2中，有一个新的 touch 命令可用于加载索引或在服务器重新启动后将文档存入内存.仅应在初次启动时使用它来热身"服务器，否则可能会无助于将实际的活动"数据强行拉出内存.
In MongoDB 2.2, there is a new touch command you can use to load indexes or documents into memory after a server restart. This should only be used on initial startup to "warm up" the server, as otherwise you could be unhelpfully forcing actual "active" data out of memory.
例如，在linux系统上，您可以使用top命令，并且应该看到:
On a linux system, for example, you can use the top command and should see that:
- 虚拟字节/VSIZE往往是整个数据库的大小
- 如果服务器没有正在运行的其他进程，则常驻字节/RSIZE将是计算机的总内存(包括文件系统缓存的内容)
-
mongod不应使用swap(因为文件是内存映射的)
- virtual bytes/VSIZE will tend to be the size of the entire database
- if the server doesn't have other processes running, resident bytes/RSIZE will be the total memory of the machine (this includes file system cache contents)
mongodshould not use swap (since the files are memory-mapped)
您可以使用 mongostat 工具快速查看mongod活动.或更有效的方法是，使用 MMS 之类的服务来监控一段时间内的指标.
You can use the mongostat tool to get a quick view of your mongod activity .. or more usefully, use a service like MMS to monitor metrics over time.
MongoDB 查询优化器每执行约1,000次写入操作，就查询模式的计划执行情况进行一次比较，然后缓存获胜"查询计划，直到优化器下次运行..或者您明确调用 explain() 在该查询上.
The MongoDB Query Optimizer compares plan execution for a query pattern every ~1,000 write operations, and then caches the "winning" query plan until the next time the optimizer runs .. or you explicitly call an explain() on that query.
这应该是一个简单的测试对象:使用.explain()在mongo shell中运行查询，并查看ms计时，以及索引条目和扫描的文档数. describe()的时间并不是查询运行的实际时间，因为它包括比较计划的成本.典型的执行速度会更快..，您可以在mongod日志中查找慢查询.
This should be a straightforward one to test: run your query in the mongo shell with .explain() and look at the ms timings, and also the number of index entries and documents scanned. The timing for an explain() isn't the actual time the queries will take to run, as it includes the cost of comparing the plans. The typical execution will be much faster .. and you can look for slow queries in your mongod log.
默认情况下，MongoDB将记录所有小于100ms的查询，因此这为寻找优化查询提供了一个很好的起点.您可以使用 --slowms 配置选项来调整慢毫秒值.使用 Database Profiler 命令.
By default MongoDB will log all queries slower than 100ms, so this provides a good starting point to look for queries to optimize. You can adjust the slow ms value with the --slowms config option, or using the Database Profiler commands.
- Caching
- Checking Server Memory Usage
- Database Profiler
- Explain
- Monitoring & Diagnostics
这篇关于MongoDB将文档预加载到RAM中以提高性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
