使用存储库模式时的MongoDB和大型数据集 [英] MongoDB and Large Datasets when using a Repository pattern
问题描述
好的，因此在工作中，我们正在使用MVC C#和amp;开发系统. MongoDB.刚开始开发时，我们决定遵循Repository模式(这真是个麻烦！)可能是个好主意，下面的代码可以让您大致了解当前已实现的内容.
Okay so at work we are developing a system using MVC C# & MongoDB. When first developing we decided it would probably be a good idea to follow the Repository pattern (what a pain in the ass!), here is the code to give an idea of what is currently implemented.
MongoRepository类:
The MongoRepository class:
public class MongoRepository { }
public class MongoRepository<T> : MongoRepository, IRepository<T>
where T : IEntity
{
private MongoClient _client;
private IMongoDatabase _database;
private IMongoCollection<T> _collection;
public string StoreName {
get {
return typeof(T).Name;
}
}
}
public MongoRepository() {
_client = new MongoClient(ConfigurationManager.AppSettings["MongoDatabaseURL"]);
_database = _client.GetDatabase(ConfigurationManager.AppSettings["MongoDatabaseName"]);
/* misc code here */
Init();
}
public void Init() {
_collection = _database.GetCollection<T>(StoreName);
}
public IQueryable<T> SearchFor() {
return _collection.AsQueryable<T>();
}
}
IRepository接口类:
The IRepository interface class:
public interface IRepository { }
public interface IRepository<T> : IRepository
where T : IEntity
{
string StoreNamePrepend { get; set; }
string StoreNameAppend { get; set; }
IQueryable<T> SearchFor();
/* misc code */
}
然后使用Ninject实例化存储库，但如果没有该存储库，它将看起来像这样(只是为了使这个例子更简单):
The repository is then instantiated using Ninject but without that it would look something like this (just to make this a simpler example):
MongoRepository<Client> clientCol = new MongoRepository<Client>();
这是用于搜索页面的代码，用于馈入控制器动作，该动作为带有要读取的DataTables的表输出JSON.请注意，以下使用DynamicLinq，以便可以从字符串输入中构建linq:
Here is the code used for the search pages which is used to feed into a controller action which outputs JSON for a table with DataTables to read. Please note that the following uses DynamicLinq so that the linq can be built from string input:
tmpFinalList = clientCol
.SearchFor()
.OrderBy(tmpOrder) // tmpOrder = "ClientDescription DESC"
.Skip(Start) // Start = 99900
.Take(PageLength) // PageLength = 10
.ToList();
现在的问题是，如果集合中有很多记录(准确的说是99,905)，那么如果字段中的数据不是很大，例如我们的Key字段是5个字符的固定长度字符串，并且可以使用此查询略过和精打细算.但是，如果类似ClientDescription的内容可以更长，我可以从查询的前面(即第1页)排序"和取"罚款，但是当我使用Skip = 99900&取= 10会产生以下内存错误:
Now the problem is that if the collection has a lot of records (99,905 to be exact) everything works fine if the data in a field isn't very large for example our Key field is a 5 character fixed length string and I can Skip and Take fine using this query. However if it is something like ClientDescription can be much longer I can 'Sort' fine and 'Take' fine from the front of the query (i.e. Page 1) however when I page to the end with Skip = 99900 & Take = 10 it gives the following memory error:
发生类型为'MongoDB.Driver.MongoCommandException'的异常 在MongoDB.Driver.dll中，但未在用户代码中处理
An exception of type 'MongoDB.Driver.MongoCommandException' occurred in MongoDB.Driver.dll but was not handled in user code
其他信息:命令聚合失败:异常:排序 超出了104857600字节的内存限制，但没有选择加入 外部排序.中止操作.传递allowDiskUse:true以选择 内..
Additional information: Command aggregate failed: exception: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in..
好的，我想这很容易理解.我在线上看了看，建议的大多数内容都是使用Aggregation和"allowDiskUse:true"，但是由于我在IRepository中使用IQueryable，所以我无法开始使用IAggregateFluent<>，因为随后您需要将与MongoDB相关的类公开给IRepository违反IoC负责人.
Okay so that is easy to understand I guess. I have had a look online and mostly everything that is suggested is to use Aggregation and "allowDiskUse:true" however since I use IQueryable in IRepository I cannot start using IAggregateFluent<> because you would then need to expose MongoDB related classes to IRepository which would go against IoC principals.
是否有任何方法可以强制IQueryable使用它，或者有人知道我可以在不违反IoC主体的情况下访问IAggregateFluent的方法吗?
Is there any way to force IQueryable to use this or does anyone know of a way for me to access IAggregateFluent without going against IoC principals?
我感兴趣的一件事是为什么排序适用于第1页(开始= 0，采用= 10)，但是当我搜索到末尾时却失败了……肯定必须对所有内容进行排序才能使我能够获取第1页中的项目，但不应该(开始= 99900，获取= 10)只需要相同数量的排序"，而MongoDB应该只向我发送最后5条左右的记录.为什么两种排序都没有发生此错误?
答案
好的，因此在@ craig-wilson的帮助下，升级到最新版本的MongoDB C#驱动程序并在MongoRepository中更改以下内容将解决此问题:
Okay so with the help of @craig-wilson upgrading to the newest version of MongoDB C# drivers and changing the following in MongoRepository will fix the problem:
public IQueryable<T> SearchFor() {
return _collection.AsQueryable<T>(new AggregateOptions { AllowDiskUse = true });
}
我正在获取System.MissingMethodException，但这是由MongoDB驱动程序的其他副本也需要更新引起的.
I was getting a System.MissingMethodException but this was caused by other copies of the MongoDB drivers needing updated as well.
推荐答案
从IMongoCollection创建IQueryable时，您可以传入AggregateOptions，以允许您设置AllowDiskUse.
When creating the IQueryable from an IMongoCollection, you can pass in the AggregateOptions which allow you to set AllowDiskUse.
这篇关于使用存储库模式时的MongoDB和大型数据集的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
