实体框架的大数据集，内存溢出异常 [英] Entity framework large data set, out of memory exception

问题描述

我的工作的一个非常大的数据集，大约200万条记录。我有下面的代码，但出现内存不足的异常有围绕三个批次，60万条记录过程之后。据我所知，因为它遍历每个批次的实体框架延迟加载，那么这是试图将全2万条记录建立到内存中。 ？有什么办法来卸载一批一我已经处理它

I am working the a very large data set, roughly 2 million records. I have the code below but get an out of memory exception after it has process around three batches, about 600,000 records. I understand that as it loops through each batch entity framework lazy loads, which is then trying to build up the full 2 million records into memory. Is there any way to unload the batch one I've processed it?

ModelContext dbContext = new ModelContext();
IEnumerable<IEnumerable<Town>> towns = dbContext.Towns.OrderBy(t => t.TownID).Batch(200000);
foreach (var batch in towns)
{
    SearchClient.Instance.IndexMany(batch, SearchClient.Instance.Settings.DefaultIndex, "Town", new SimpleBulkParameters() { Refresh = false });
}

注：批方法来源于此项目：的 https://code.google.com/p/morelinq/

搜索客户端是这样的： https://github.com/Mpdreamz/NEST

The search client is this: https://github.com/Mpdreamz/NEST

推荐答案

的问题是，当你从EF获取数据其实有创建数据的两个副本，其中一个被返回到EF持有到并使用更改的用户和第二检测（以便它可以持续改变到数据库）。 EF持有的背景和它的这一套，多数民众赞成的生命周期第二组运行你的内存不足。

The issue is that when you get data from EF there are actually two copies of the data created, one which is returned to the user and a second which EF holds onto and uses for change detection (so that it can persist changes to the database). EF holds this second set for the lifetime of the context and its this set thats running you out of memory.

您有2个选项来处理这个

You have 2 options to deal with this

1. 更新您的上下文每批



2. 使用.AsNoTracking（）在查询例如：

1. renew your context each batch

2. Use .AsNoTracking() in your query eg:

IEnumerable<IEnumerable<Town>> towns = dbContext.Towns.AsNoTracking().OrderBy(t => t.TownID).Batch(200000);

这告诉EF不保留副本变化检测。你可以阅读更多有关什么AsNoTracking确实和这个在我的博客对性能的影响：的http：// blog.staticvoid.co.nz/2012/4/2/entity_framework_and_asnotracking

this tells EF not to keep a copy for change detection. You can read a little more about what AsNoTracking does and the performance impacts of this on my blog: http://blog.staticvoid.co.nz/2012/4/2/entity_framework_and_asnotracking

这篇关于实体框架的大数据集，内存溢出异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！