FORTRAN:存储可在MATLAB中读取的大量数据的最佳方法 [英] FORTRAN: Best way to store large amount of data which is readable in MATLAB

问题描述

我正在使用Fortran开发应用程序，其中有一些点在对象表面上定义四边形面板.我正在这些四边形面板上针对许多频率计算各种参数.

I am working on developing an application in Fortran where I have points defining quadrilateral panels on the surface of an object. I am calculating various parameters on these quadrilateral panels for a number of frequencies.

输出文件应如下所示:

FREQUENCY,PANEL_NUMBER,X1,Y1,Z1,X2,Y2,Z2,X3,Y3,Z3,X4,Y4,Z4,AREA,PRESSURE,....
0.01,1,....
0.01,2,....
0.01,3,....
.
.
.
.
0.01,2000,....
0.02,1,....
0.02,2,....
.
.
.
0.02,2000,...
.
.

我期望最多300,000行和30列.数据类型由整数，实数和复数组成.我要存储此文件，以后再在MATLAB中读取该文件以创建3D几何图形，我将根据每个面板上的压力对其进行着色.

I am expecting a maximum of 300,000 rows with 30 columns. Data types are composed of integer, real and complex numbers. I want to store this file and later read the file in MATLAB to create a 3D geometry which I will color based on pressure at each panel.

问题是，如从文件结构中可以看到的，有很多数据.我目前正在将此文件编写为CSV文件，大小约为26GB.

The problem is, as you can see from the file structure, there is lot of data. I am currently writing this as a CSV file and the size is about 26GB.

我不想使用数据库来处理此问题.谁能建议我应该使用FORTRAN将此文件写入哪种文件格式.

I do not want to use database to handle this. Could anyone suggest what file format I should write this data using FORTRAN.

感谢您的帮助， 阿米塔瓦

Thanks for your help, Amitava

推荐答案

以计算机的原始格式存储数据，而不是将数字转换为以10为底的字符的人类可读文件.这将产生最小的文件，并且处理最快.在Fortran open语句上，使用form='unformatted', access='stream'.第一种导致文件未格式化，第二种导致Fortran不包括其通常的记录长度信息，该信息是Fortran特定的.此遗漏使文件更易于移植到其他语言.其他人可以更好地帮助您如何在MATLAB中读取文件.我在网上找到了它: http://www.mathworks.com/help/matlab/import_export/importing-binary-data-with-low-level-io.html

Store the data in the native format of the computer rather than in a human-readable file in which the numbers have been converted to base 10 and characters. This will produce the smallest file and the fastest to process. On the Fortran open statement, use form='unformatted', access='stream'. The first causes the file to be unformatted, the second causes Fortran not to include its usual record-length information, which is Fortran specific. This omission makes the file more portable to other languages. Someone else can help better with how to read the file in MATLAB; I found this on the web: http://www.mathworks.com/help/matlab/import_export/importing-binary-data-with-low-level-i-o.html

更新:此方法有几个假设.如果您希望在不同类型的计算机之间传输文件，则可能无法轻松工作.您的问题意味着要包含许多相同内容的行.相同的行仅将具有相同数量记录的文件结构匹配.似乎您想读取整个文件，在这种情况下，顺序文件是合适的.如果您希望读取随机"记录，则Fortran直接访问文件可能会很有用.由于相同记录的简单性，使用本机文件格式似乎很容易.如果您希望在计算机之间进行自我文档化或可移植性(不同的数字表示形式)，则HDF或FITS之类的文件格式将非常有用.

UPDATE: This approach has several assumptions. It might not work easily if you wish to transport the file between different types of computers. Your question implies that want many rows of identical content. Identical rows simply matches a file structure with that number of identical records. It seems that you want to read the entire file, in which case a sequential file is appropriate. If you wish to read "random" records, a Fortran direct access file might be useful. With the simplicity of identical records, using a native file format seems easy. If you want self-documentation or portability across computers (different numeric representations), a file format such as HDF or FITS would be useful.

这篇关于FORTRAN:存储可在MATLAB中读取的大量数据的最佳方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！