了解并发文件从多个进程写入 [英] Understanding concurrent file writes from multiple processes

问题描述

从这里开始：原子是文件追加原子在UNIX

考虑多个进程打开相同的文件并追加给它的案件。 O_APPEND保证谋求文件的末尾，然后开始写操作是原子。因此，多个进程可以附加到同一个文件，没有过程将尽量覆盖所有其它进程写，因为每个写入大小为＆lt; = PIPE_BUF

我写了一个测试程序，其中多个进程打开和写入同一个文件（写（2））。我要确保每次写的大小是> PIPE_BUF（4K）。我期待看到一个过程覆盖别人的数据实例。但是，这并不发生。我用不同的写大小进行测试。是不是只是运气还是有一个原因，那不会发生？
我的最终目标是理解，如果多个进程追加到同一文件中需要协调的写操作。

下面是完整的程序。每一个进程创建一个int缓冲区，填充所有的值，其等级，打开一个文件，并写入到它。

规格：
的openmpi 1.4.3
openSUSE的11.3 64位

编译为：mpicc -O3 test.c的，
运行如下：-np的mpirun 8 ./a.out

 的#include＆LT;＆stdio.h中GT;
＃包括LT＆;＆stdlib.h中GT;
＃包括LT＆;＆mpi.h GT;
＃包括LT＆;＆fcntl.h GT;
＃包括LT＆;＆unistd.h中GT;
＃包括LT＆; SYS / types.h中＆GT;
＃包括LT＆; SYS / stat.h＆GT;INT
主（INT ARGC，字符** argv的）{
    INT等级，尺寸，我，BUFSIZE = 134217728，FD，状态= 0，bytes_written，tmp_bytes_written;
    INT * BUF;
    字符*文件名=/tmp/testfile.out;    MPI_INIT（安培; ARGC，＆安培; argv的）;
    MPI_Comm_rank（MPI_COMM_WORLD，＆安培;等级）;
    MPI_Comm_size（MPI_COMM_WORLD，＆安培;大小）;    BUF =（INT *）malloc的（BUFSIZE *的sizeof（INT））;
    如果（BUF == NULL）{
        状态= -1;
        PERROR（无法malloc的）;
        转到最后确定;
    }
    对于（i = 0; I＆LT; BUFSIZE;我++）
        BUF [I] =排名;    如果（-1 ==（FD =开放（文件名，O_APPEND | O_WRONLY，S_IWUSR）））{
        PERROR（无法打开文件）;
        状态= -1;
        转到结束;
        出口（-1）;
    }    bytes_written = 0;
    如果（BUFSIZE！=（tmp_bytes_written =写（FD，BUF，BUFSIZE）））{
        PERROR（中写入错误）;
        的printf（RET值数：％d \\ n，tmp_bytes_written）;
        状态= -1;
        转到关闭;
    }关：
    如果（-1 ==接近（FD））{
        PERROR（错误时合）;
        状态= -1;
    }
结束：
    免费（BUF）;
最终确定：
    MPI_Finalize（）;
    返回状态;
}
 

解决方案

比 PIPE_BUF 写少的原子仅适用于管道和FIFO。对于文件的写入，POSIX说：

  

POSIX.1-2008的本卷未指定的并发行为
  写入从多个进程的文件。应用程序应该使用一些
  形式的并发控制。

...这意味着你对你自己的 - 不同的UNIX的喜好会得出不同的保障

From here : Is file append atomic in UNIX

Consider a case where multiple processes open the same file and append to it. O_APPEND guarantees that seeking to the end of file and then beginning the write operation is atomic. So multiple processes can append to the same file and no process will overwrite any other processes' write as far as each write size is <= PIPE_BUF.

I wrote a test program where multiple processes open and write to the same file (write(2)). I make sure each write size is > PIPE_BUF (4k). I was expecting to see instances where a process overwrites someone else's data. But that doesnt happen. I tested with different write sizes. Is that just luck or is there a reason why that doesn't happen? My ultimate goal is to understand if multiple processes appending to the same file need to co-ordinate their writes.

Here is the complete program. Every process creates an int buffer, fills all values with its rank, opens a file and writes to it.

Specs: OpenMPI 1.4.3 on Opensuse 11.3 64-bit

Compiled as: mpicc -O3 test.c, run as: mpirun -np 8 ./a.out

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>

int 
main(int argc, char** argv) {
    int rank, size, i, bufsize = 134217728, fd, status = 0, bytes_written, tmp_bytes_written;
    int* buf;
    char* filename = "/tmp/testfile.out";

    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    buf = (int*) malloc (bufsize * sizeof(int));   
    if(buf == NULL) {
        status = -1;
        perror("Could not malloc");
        goto finalize;
    }
    for(i=0; i<bufsize; i++) 
        buf[i] = rank;

    if(-1 == (fd = open(filename, O_APPEND|O_WRONLY, S_IWUSR))) {
        perror("Cant open file");
        status = -1;
        goto end;
        exit(-1);
    }

    bytes_written = 0;
    if(bufsize != (tmp_bytes_written = write(fd, buf, bufsize))) {
        perror("Error during write");
        printf("ret value: %d\n", tmp_bytes_written);
        status = -1;
        goto close;
    }

close:
    if(-1 == close(fd)) {
        perror("Error during close");
        status = -1;
    }
end:
    free(buf);
finalize:
    MPI_Finalize();
    return status;
}

解决方案

Atomicity of writes less than PIPE_BUF applies only to pipes and FIFOs. For file writes, POSIX says:

This volume of POSIX.1-2008 does not specify behavior of concurrent writes to a file from multiple processes. Applications should use some form of concurrency control.

...which means that you're on your own - different UNIX-likes will give different guarantees.

这篇关于了解并发文件从多个进程写入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！