如何使用c＃windows应用程序读取10GB .csv文件 [英] How to read 10GB .csv file using c# windows application

问题描述

我尝试读取10GB .csv文件并导入我们的数据库。但是当我通过数据适配器读取此文件来填充数据集或数据表时，它会给出内存异常。



请帮助我....

I am try to read 10GB .csv file and import in our db. but when i am read this file via data adapter to fill data set or data table its give out of memory exception.

Please Help Me....

推荐答案

常识应该告诉你，你无法一次性阅读所有内容。以较小的块读取它并单独处理每个块，直到到达输入的末尾。

Common sense should tell you that you cannot read all that in one go. Read it in smaller chunks and process each chunk in isolation until you reach the end of the input.

10GB CSV对于数据存储是相当紧凑的：一旦它被读入单个DataColumn DataRows中的值会变得相当大，因为CSV行中的每个值都将作为一个对象保存，并且需要所有开销。



因此，您唯一的选择是读取文件（以块的形式放入可重用的内存区域或逐行）并根据需要处理每个文件。尝试将这么多数据读入一个疙瘩只是愚蠢 - 你当然不能把它呈现给用户，这几乎是做这件事的唯一原因。

A 10GB CSV is a fairly compact for of data storage: once it's been read into individual DataColumn values within DataRows however it is going to be considerably bigger, as each value from the CSV line is going to be held as an object with all the overheads that requires.

So your only option is to read the file (either in chunks into a reusable memory area, or line-by-line) and process each as you need it. Attempting to read that much data into anything in one "lump" is just silly - you certainly can't present it to the user which is pretty much the only reason for doing it.

这篇关于如何使用c＃windows应用程序读取10GB .csv文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！