使用 MPI-IO 读取文本文件? [英] Reading text files using MPI-IO?

问题描述

我有一个文本文件，其中包含一个带有矩阵维度的标题，然后是矩阵.以下是 3x3 矩阵的示例:

I have a text file which contains a header with the dimensions of a matrix, and then the matrix. Here's an example for a 3x3 matrix:

3 3
1 56 8
12 3 0
9 44 81

我一直使用 MPI-IO 获取垃圾值，并发现它仅适用于二进制文件，而不适用于文本文件.

I kept getting garbage values with MPI-IO, and found out it only works with binary, not text files.

我以为我会读入字符流并转换为整数，但我不确定如何解决这个问题，因为矩阵元素的位数是可变的.我真的不知道如何处理这个问题?

I thought I'd read in a character stream and convert to integers, but I'm not sure how to approach this problem since the matrix elements have variable number of digits. I'm really not sure how to approach this?

推荐答案

通常你知道二进制文件中写入的东西的类型和数量(例如所有整数，10 int 3 浮动等).您可以读取字节数，但 MPI 二进制文件通常作为整数类型读取/写入，在您的情况下为 9 个整数(因此位数并不重要).您打开文件并阅读类似的内容，

Generally you know the type and number of the things written in your binary file (e.g. all integers, 10 int 3 float, etc). You can read as numbers of bytes but MPI binary files are usually read/written as a whole number of types, in your case 9 integers (so number of digits are not important). You open the file and read with something like,

    call MPI_FILE_OPEN(MPI_COMM_WORLD, filename, & 
                   MPI_MODE_RDONLY , MPI_INFO_NULL, fh, ierr)

    bufsize = 3*3
    allocate(buf(bufsize))
    call MPI_FILE_READ_ALL(fh, buf, bufsize, MPI_integer, & 
                           MPI_STATUS_IGNORE, ierr)

对于可变矩阵大小，您可以使用诸如 MPI_File_get_size 之类的东西来获取大小并计算出要读取的元素数量.对于混合数据，您可以将二进制文件的第一个(或最后一个)部分作为标头，您首先读取它并使用它来解码文件的其余部分.您仍然需要知道标头的格式，这可能会出现问题，因为您在更改代码/标头格式时会破坏向后兼容性.这是 HDF5 等数据格式的部分原因 https://support.hdfgroup.org/HDF5/

For variable matrix sizes, you can use things like MPI_File_get_size to get the size and work out how many elements to read. For mixed data, you can have the first (or last) part of the binary file as a header, which you read first and use to decode the rest of your file. You would still need to know the format of the header and this can be problematic as you break backwards compatibility when you change the code/header format. This is part of the reason for data formats like HDF5 https://support.hdfgroup.org/HDF5/

这篇关于使用 MPI-IO 读取文本文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！