MongoDB GridFS文件大小非常大，相对较小的文件 [英] MongoDB GridFS File Sizes huge for relatively small file
问题描述
我正在做一些测试，以查看是否可以在MongoDB上使用GridFS来存储将来的应用程序的文件.我正在使用10gen的C#驱动程序将80Mb文件上传"到数据库.
I'm doing some tests to see whether we can use GridFS on MongoDB to store files for a future application; I'm using 10gen's C# driver to "Upload" an 80Mb file onto the database.
第一次添加很好，大约用了3秒钟，对我的测试机器来说还算不错.但是，以后添加同一文件要花费更长的时间，最多30秒，最终MongoDB告诉我该文件内存不足并崩溃了.
The first addition was fine and took approx 3 seconds which isn't too bad on my test machine; however future additions of the same file took much longer, up to 30 seconds eventually MongoDB told me it ran out of memory and crashed out.
添加10个文件，大小为80Mb会导致在系统崩溃之前为我的数据库创建了8个文件，名为dbaseName.0到dbaseName.7，其文件大小从16Mb到512Mb呈指数增长，从文件0到5，然后是文件6和7个都为512Mb.
Adding 10 files, 80Mb in size results in 8 files being created for my database before the system crashes named dbaseName.0 to dbaseName.7 with their file sizes increasing exponentially from 16Mb to 512Mb from files 0 to 5 then files 6 and 7 are both 512Mb.
这些文件不到2Gb，显然第十次添加文件会使dbase超过2Gb，这超出了我的32位测试版本的限制.
Those files come to just under 2Gb, obviously adding the file for the 10th time takes the dbase to over 2Gb which is beyond my 32bit test version's limit.
为什么存储800Mb的文件会占用2Gb以上的空间?我在某个地方错过了一个设置吗?
Why does storing 800Mb worth of files take over 2Gb? Is there a setting I've missed somewhere?
MongoDB是否始终将整个GridFS保持在RAM中?如果是这样，磁盘的意义是什么?如果我的生产服务器上只有32Gb的RAM，那么我只能在GridFS中存储32Gb的内存吗?
Does MongoDB hold the entire GridFS in RAM constantly? If so what's the point of the disk? If I've only got 32Gb of RAM on my production server can I only store 32Gb in GridFS?
我在MongoGridFS对象上使用了sureIndexes，并检查了显示为GridFS创建索引的datbase，因此确定Mongo不应尝试将整个数据存储区放入RAM吗?
I used EnsureIndexes on my MongoGridFS object and I checked the datbase which shows indexes were created for GridFS so surely Mongo shouldn't try and fit the whole datastore into RAM?
MongoDB可以满足我们的所有需求，但是我们需要它能够容纳大型文件集合；我缺少明显的东西吗?
MongoDB fits all of our needs, but we need it to be able to hold a large file collection; am I missing something obvious?
堆栈跟踪:
Mon Oct 15 11:57:15 [conn15] insert busyNow.fs.chunks keyUpdates:0 locks(micros) w:112892 113ms
Mon Oct 15 11:57:15 [conn15] MapViewOfFileEx for /data/db/busyNow.7 failed with errno:8 Not enough storage is available to process this command. (file size is 536608768) in MemoryMappedFile::map
Mon Oct 15 11:57:15 [conn15] busyNow.fs.chunks Fatal Assertion 16166
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\util\assert_util.cpp(124) mongo::fassertFailed+0x75
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\util\mmap_win.cpp(211) mongo::MemoryMappedFile::map+0x4ce
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\db\mongommf.cpp(182) mongo::MongoMMF::create+0xa3
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\db\pdfile.cpp(469) mongo::MongoDataFile::open+0x141
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\db\database.cpp(280) mongo::Database::getFile+0x34f
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\db\database.cpp(332) mongo::Database::suitableFile+0x129
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\db\database.cpp(359) mongo::Database::allocExtent+0x41
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\db\pdfile.cpp(1271) mongo::outOfSpace+0x107
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\db\pdfile.cpp(1293) mongo::allocateSpaceForANewRecord+0x5d
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\db\pdfile.cpp(1463) mongo::DataFileMgr::insert+0x493
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\db\pdfile.cpp(1217) mongo::DataFileMgr::insertWithObjMod+0x33
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\db\instance.cpp(761) mongo::checkAndInsert+0x72
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\db\instance.cpp(821) mongo::receivedInsert+0x4cd
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\db\instance.cpp(434) mongo::assembleResponse+0x62a
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\db\db.cpp(192) mongo::MyMessageHandler::process+0xe8
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\mongo\util\net\message_server_port.cpp(86) mongo::pms::threadRun+0x424
Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\third_party\boost\boost\thread\detail\thread.hpp(62) boost::detail::thread_data<boost::_bi::bind_t<void,void (__cdecl*)(mongo::MessagingPort *),boost::_bi::list1<boost::_bi::value<mongo::MessagingPort *
> > > >::run+0x9Mon Oct 15 11:57:17 [conn15] mongod.exe ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(16707566) boost::`anonymous namespace'::thread_start_function+0x47
Mon Oct 15 11:57:17 [conn15] mongod.exe f:\dd\vctools\crt_bld\self_x86\crt\src\threadex.c(314) _callthreadstartex+0x1b
Mon Oct 15 11:57:17 [conn15] mongod.exe f:\dd\vctools\crt_bld\self_x86\crt\src\threadex.c(292) _threadstartex+0x64
Mon Oct 15 11:57:17 [conn15]
***aborting after fassert() failure
Mon Oct 15 11:58:33 [initandlisten] connection accepted from 127.0.0.1:56308 #16 (3 connections now open)
推荐答案
好；经过大量搜索之后，似乎MongoDB在指数大小的文件中预分配了高达2Gb的空间，之后每个文件将为2G.
Ok; after much searching it seems that MongoDB pre-allocates space in the exponential sized files up to 2Gb after that each file will be 2G.
http://www.mongodb.org/display/DOCS/Excessive+磁盘+空间
我的测试程序在后台文件(.0-.7等)中添加了80Mb文件，并且随着数据块开始写入最后一个文件，Mongo预先分配了比最后一个更大的文件.
My test program adds the 80Mb files, within the background files (.0 - .7 etc) and as the data chunks start to be written into the last file Mongo preallocates another file exponentially bigger than the last.
因此，第一个80Mb文件会填充16Mb文件，32Mb文件和64Mb背景文件，并且由于元数据会占用更多空间，并且必须稍微侵占128Mb文件，这会触发mongo预先分配一个256Mb文件总计496Mb;随着添加更多文件，更多文件将被预分配，并且在我的测试机上命中2Gb时，Mongo无法访问该空间并崩溃.
So the first 80Mb file, fills up the 16Mb file, the 32Mb file and the 64Mb background files and due to the meta data takes up a bit more space and must encroach slightly onto the 128Mb file, this triggers mongo to preallocate a 256Mb file totalling 496Mb; as more files are added more files are preallocated and when 2Gb is hit on my test machine Mongo can't access the space and collapses.
因此，尽管看起来一个80Mb文件占用的空间比应该占用的空间大得多-但这种处理方式是合理的.
So although it seems one 80Mb file takes up a lot more space than it should - it makes sense in a roundabout way.
这可以通过使用--noprealloc运行mongod来关闭，尽管仅建议在测试计算机上使用.
This can be turned off by running mongod with --noprealloc though this is recommended for test machines only.
感谢您的答复！
这篇关于MongoDB GridFS文件大小非常大，相对较小的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
