在MATLAB中使用一个大的CSV文件 [英] Working with a big CSV file in MATLAB

问题描述

我必须使用大型CSV档案，最高可达2GB。更具体地说，我必须将所有这些数据上传到mySQL数据库，但在我不得不做一些计算之前，所以我需要做所有这一切在MATLAB（也是我的主管想做在MATLAB，因为他熟悉只是与

解决方案

任何想法如何处理这些大文件？ div>

您应该可以使用 textscan 读取数据例如，如果你有3列数据，你可以这样做：

  filename ='fname.csv'; 
 [fh，errMsg] = fopen（filename，'rt'）; 
 if fh == -1，error （'cann''t打开文件：％s：％s'，filename，errMsg）; end 
 N = 100;％一次读取100行
 while〜feof（fh）
c = textscan（fh，'％f％f％f'，N，'Delimiter'，'，'）; 
 doStuff（c）; 
 end 
  

EDIT

这些天（R2014b及更高版本），更容易，请使用 数据存储 。

I have to work with a big CSV file, up to 2GB. More specifically I have to upload all this data to the mySQL database, but before I have to make a few calculation on that, so I need to do all this thing in MATLAB (also my supervisor want to do in MATLAB because he familiar just with MATLAB :( ).

Any idea how can I handle these big files?

解决方案

You should probably use textscan to read the data in in chunks and then process. This will probably be more efficient than reading a single line at a time. For example, if you have 3 columns of data, you could do:

filename = 'fname.csv';
[fh, errMsg] = fopen( filename, 'rt' );
if fh == -1, error( 'couldn''t open file: %s: %s', filename, errMsg ); end
N  = 100; % read 100 rows at a time
while ~feof( fh )
  c  = textscan( fh, '%f %f %f', N, 'Delimiter', ',' );
  doStuff(c);
end

EDIT

These days (R2014b and later), it's easier and probably more efficient to use a datastore.

这篇关于在MATLAB中使用一个大的CSV文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！