DataTable内存消耗巨大 [英] DataTable memory huge consumption
问题描述
我正在将csv数据从文件加载到数据表中进行处理.
I´m loading csv data from files into a datatable for processing.
问题是，我想处理多个文件，而对数据表的测试显示了我巨大的内存消耗 我测试了一个37MB的csv文件，并且内存增长到了240MB，这是很多恕我直言的方式. 我读到，数据表中有开销，我可以承受大约70MB的大小，而不是240MB，这意味着它是原始大小的六倍. 我在这里读到，数据表比POCO需要更多的内存，但是区别太大了.
The problem is, that I want to process several files and my tests with the datatable shows me huge memory consumption I tested with a 37MB csv file and the memory growed up to 240MB, which is way to much IMHO. I read, that there is overhead in the datatable and I could live with about 70MB in size , but not 240MB, which means it is six times the original size. I read here, that datatables need more memory than POCOs, but that the difference is way too much.
我放了一个内存探查器，看了一下是否有内存泄漏以及内存在哪里.我发现，datatable列有6MB到19MB之间的字符串填充，而datatable约有20列.值存储在列中吗?为什么要占用这么多的内存，我该怎么做以减少内存消耗. 有了这种内存消耗，数据表似乎无法使用.
I put on a memory profiler and looked, if I have memory leaks and where the memory is. I found, that the datatablecolumns have between 6MB and 19MB filled with strings and the datatable had about 20 columns. Are the values stored in the columns? Why is so much memory taken, what can I do to reduce memory consumption. With this memory consumption datattables seem to be unusable.
是否有人在数据表中遇到了此类问题，或者我做错了什么?
Had somebody else such problems with datatables, or I´m doing something wrong?
PS:我尝试了一个70MB的文件，数据表增长到了500MB！
PS: I tried a 70MB file and the datatable growed up to 500MB!
好的，这是一个小测试用例: 37MB的csv文件(21列)使内存增长到179MB.
OK here is a small testcase: The 37MB csv-file (21 columns) let the memory grow up to 179MB.
private static DataTable ReadCsv()
{
DataTable table = new DataTable();
table.BeginLoadData();
using (var reader = new StreamReader(File.OpenRead(@"C:\Develop\Tests\csv-Data\testdaten\test.csv")))
{
int y = 0;
int columnsCount = 0;
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
var values = line.Split(',');
if (y == 0)
{
columnsCount = values.Count();
// create columns
for (int x = 0; x < columnsCount; x++)
{
table.Columns.Add(new DataColumn(values[x], typeof(string)));
}
}
else
{
if (values.Length == columnsCount)
{
// add the data
table.Rows.Add(values);
}
}
y++;
}
table.EndLoadData();
table.AcceptChanges();
}
return table;
}
推荐答案
DataSet及其子元素DataTable，DataRow等构成内存中的关系数据库.涉及很多开销(尽管确实使[某些]事情非常方便.
DataSet and its children DataTable, DataRow, etc. make up an in-memory relational database. There is a lot of overhead involved (though it does make [some] things very convenient.
如果内存有问题，
- 构建域对象以表示具有类型属性的CSV文件中的每一行.
- 创建一个自定义集合(或仅使用
IList<T>来保存它们
或者，使用 - 通过数字选择行的功能
- 能够按行号和列名或列号在一行中选择一列.
- 了解列名称的有序集合的能力
- 奖金:能够按名称或序号选择一列并接收其值的列表，每行一个.
- Build domain objects to represent each row in your CSV file with typed properties.
- Create a custom collection (or just use
IList<T>to hold them - Alternatively, build a light-weight class with the basic semantics of a
DataTable:- the ability to select a row by number
- the ability to select a column within a row by row number and either column name or number.
- The ability to know the ordered set of column names
- Bonus: The ability to select a column by name or ordinal number and receive a list of its values, one per row.
确定要在内存中显示CSV文件吗?您可以通过
IDataReader来访问它们吗，例如Sebastien Lorion的快速CSV阅读器?Are you sure you need an in-memory representation of your CSV files? Could you access them via an
IDataReaderlike Sebastien Lorion's Fast CSV Reader?这篇关于DataTable内存消耗巨大的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
DataTable的基本语义来构建轻量级类:
