DataTables上的OutOfMemoryException [英] OutOfMemoryException on DataTables
问题描述
在我的应用程序中，我有文本（平面）文件作为输入，我必须生成
一个XML文件。最大输入文本文件大小可以是900MB，并且生成xml
可能会导致2+ GB。
根据文本文件中的第一列值，行将被移动到
在飞行中创建的23个DataTable中的任何一个。
例如。
01; - 数据表1-
02; - 表2的数据
03; - 表3-的数据
04; - 表的数据4-
04; - 表4-
04的数据; - 表4的数据
04; - 表4的数据
04; - 表4的数据
基于子关系，我必须在剥离
$ b后创建XML节点$ b父子关系列。
我正在使用XmlTextWriter来编写xml数据，这些数据是根据具有不同条件的select从
表中获取的。
服务器使用2GB RAM内存
当应用程序处理500MB文件时，它会抛出OutOfMemoryException。
在任务中ager，系统抛出时使用的内存
OutOfMemoryException是2.15 GB。
但在我的桌面上，出乎意料的是，系统有512 MB，当时我加载了几个应用程序的
，进程内存达到1GB而没有任何
OutOfMemoryException问题。
你能帮助我吗？尊重。
提前致谢
问候
venkatg
Hi,
In my application I have text(flat) file as input and I have to generate
an XML file. The maximum input text file size can be 900MB and gererated xml
may result 2+ GB.
Based on the first column value from the text file, the row will be moved to
any of those 23 DataTable which are created onfly.
For eg.
01;- data for Table 1-
02;- data for Table 2-
03;- data for Table 3-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
Based on the child relation, I have to create XML node, after stripping the
parent-child relation column.
I am using XmlTextWriter to write the xml data, which are fetched from
tables based on select with different condition.
The server uses 2GB RAM memory
When the application process a 500MB file, it throws OutOfMemoryException.
In the task manager, the utilised memory when the system throws
OutOfMemoryException is 2.15 GB.
But in my desktop, for the surprise, the system has 512 MB and when I load
several application, the process memory reached 1GB without any
OutOfMemoryException issues.
Can you one help me in this regard.
Thanks in advance
With Regards
venkatg
推荐答案
Venkatachalam< Ve *********** @ discussion.microsoft.com>写道：
Venkatachalam <Ve***********@discussions.microsoft.com> wrote:
在我的应用程序中，我有文本（平面）文件作为输入，我必须生成一个XML文件。最大输入文本文件大小可以是900MB，并且生成的xml
可能会导致2+ GB。
你真的需要同时在内存中保存所有内容吗？它听起来好像更有可能工作如果你可以处理
文件写出数据，即使你需要处理
输入文件几次。
< snip>
当应用程序处理500MB文件时，它会抛出OutOfMemoryException。
在任务管理器中，系统抛出时使用的内存
OutOfMemoryException为2.15 GB。
我相信如果没有一些额外的支持（我不知道
的详细信息），单个进程不能超过2GB。
但是在我的桌面上，出乎意料的是，系统有512 MB，当我加载几个应用程序时，进程内存达到1GB而没有任何问题OutOfMemoryException问题。
你能在这方面帮助我。
In my application I have text(flat) file as input and I have to generate
an XML file. The maximum input text file size can be 900MB and gererated xml
may result 2+ GB.
Do you really need to hold all of that in memory at the same time? It
sounds like it would be much more likely to work if you could process
the file writing out data as you went, even if you need to process the
input file a few times.
<snip>
When the application process a 500MB file, it throws OutOfMemoryException.
In the task manager, the utilised memory when the system throws
OutOfMemoryException is 2.15 GB.
I believe that without some extra support (which I don''t know the
details of) a single process can''t go above 2GB.
But in my desktop, for the surprise, the system has 512 MB and when I load
several application, the process memory reached 1GB without any
OutOfMemoryException issues.
Can you one help me in this regard.
这是你的桌面交换内存到磁盘。请注意，当这个
发生时，任何试图使用该虚拟内存的东西都会大大减慢
。
-
Jon Skeet - < sk *** @ pobox.com>
http://www.pobox.com/~skeet 博客： http://www.msmvps.com/jon.skeet
如果回复小组，请不要给我发邮件
That''s your desktop swapping memory out to disk. Note that when this
happens, anything which tries to use that virtual memory will slow down
hugely.
--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
可能你的家用电脑启用了虚拟内存，其大小为
分配给超过512 MB的东西，但你的服务器电脑没有。
无论如何，当你处理大量数据时，你不应该使用
DataSet或DataTable或类似的东西，因为那些数据对象
会将所有数据预加载到记忆。你应该使用一些基于光标的
只转发Reader类来按需读取和处理数据。
Sheva
" Venkatachalam" < Ve的*********** @ discussions.microsoft.com>在消息中写道
news：81 ********************************** @ microsof t.com ...
Probably your home computer has virtual memory enabled, and its size is
allocated to something more than 512 MB, but your server computer doesn''t.
Anyway, when you process a large quantity of data, you should not use
DataSet or DataTable or sorta things like that, because those data objects
will preload all the data into memory. you should use some cursor based
forward only Reader classes to read and process the data on demand.
Sheva
"Venkatachalam" <Ve***********@discussions.microsoft.com> wrote in message
news:81**********************************@microsof t.com...
在我的应用程序中，我有文本（平面）文件作为输入，我必须生成一个XML文件。最大输入文本文件大小可以是900MB，并且可以生成2GB以上的数据。
基于文本文件中的第一列值，该行将被移动
对于例如。
01; - 表1-的数据
02; - 数据表2-
03; - 表3-
04的数据; - 表4-
04的数据; - 表4-
04的数据; - 表的数据4-
04; - 表4的数据
04; - 表4的数据
基于子关系，我必须在剥离后创建XML节点
亲子关系栏。
我正在使用XmlTextWriter来编写xml数据，这些数据是根据不同条件的select从
表中提取的。服务器使用2GB RAM内存
当应用程序处理500MB文件时，它会抛出OutOfMemoryException。
在任务管理器中，系统抛出时使用的内存/> OutOfMemoryException是2.15 GB。
但是在我的桌面上，出乎意料的是，系统有512 MB，当我加载几个应用程序时，进程内存达到了1GB，没有任何问题OutOfMemoryException问题。
你能在这方面帮助我吗。
提前致谢
关心
venkatg
Hi,
In my application I have text(flat) file as input and I have to generate
an XML file. The maximum input text file size can be 900MB and gererated
xml
may result 2+ GB.
Based on the first column value from the text file, the row will be moved
to
any of those 23 DataTable which are created onfly.
For eg.
01;- data for Table 1-
02;- data for Table 2-
03;- data for Table 3-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
04;- data for Table 4-
Based on the child relation, I have to create XML node, after stripping
the
parent-child relation column.
I am using XmlTextWriter to write the xml data, which are fetched from
tables based on select with different condition.
The server uses 2GB RAM memory
When the application process a 500MB file, it throws OutOfMemoryException.
In the task manager, the utilised memory when the system throws
OutOfMemoryException is 2.15 GB.
But in my desktop, for the surprise, the system has 512 MB and when I load
several application, the process memory reached 1GB without any
OutOfMemoryException issues.
Can you one help me in this regard.
Thanks in advance
With Regards
venkatg
所有数据都必须在数据库中，因为表中涉及非常复杂的关系
。
如果我拆分文件进行读取，那么性能将会非常差。
服务器还有4个Intel Xeon CPU 3.00GHz处理器。
关于
Venkat
" Jon Skeet [C＃MVP]"写道：
All the data has to be in the momory, as there are very complex relation
involved among the tables.
If I split the file to read, then the performance will go ver very worst.
And also the server has 4 Intel Xeon CPU 3.00GHz processor.
With regards
Venkat
"Jon Skeet [C# MVP]" wrote:
Venkatachalam< Ve *********** @ discussion.microsoft.com>写道：
Venkatachalam <Ve***********@discussions.microsoft.com> wrote:
在我的应用程序中，我有文本（平面）文件作为输入，我必须生成一个XML文件。最大输入文本文件大小可以是900MB，格式化xml可能会导致2+ GB。
In my application I have text(flat) file as input and I have to generate
an XML file. The maximum input text file size can be 900MB and gererated xml
may result 2+ GB.
你真的需要同时在内存中保存所有内容吗？如果您可以处理文件写出数据，即使您需要处理
输入文件几次，这听起来更有可能工作。
< snip>
Do you really need to hold all of that in memory at the same time? It
sounds like it would be much more likely to work if you could process
the file writing out data as you went, even if you need to process the
input file a few times.
<snip>
当应用程序处理一个500MB的文件时，它会抛出OutOfMemoryException。
在任务管理器中，当系统使用内存时抛出
OutOfMemoryException是2.15 GB。
When the application process a 500MB file, it throws OutOfMemoryException.
In the task manager, the utilised memory when the system throws
OutOfMemoryException is 2.15 GB.
我相信没有一些额外的支持（我不知道
的详细信息）单个进程可以'超过2GB。
I believe that without some extra support (which I don''t know the
details of) a single process can''t go above 2GB.
但在我的桌面上，出乎意料的是，系统有512 MB，当我加载几个应用程序时，进程内存达到1GB而没有任何
OutOfMemoryException问题。
你能在这方面帮助我。
But in my desktop, for the surprise, the system has 512 MB and when I load
several application, the process memory reached 1GB without any
OutOfMemoryException issues.
Can you one help me in this regard.
这是你的桌面将内存交换到磁盘。请注意，当这种情况发生时，任何试图使用该虚拟内存的东西都会大大减慢。
-
Jon Skeet - < sk *** @ pobox.com>
http://www.pobox.com/~skeet 博客： http://www.msmvps.com/jon.skeet
如果回复小组，请不要给我发邮件
That''s your desktop swapping memory out to disk. Note that when this
happens, anything which tries to use that virtual memory will slow down
hugely.
--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
这篇关于DataTables上的OutOfMemoryException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
