了解来自多个进程的并发文件写入 [英] Understanding concurrent file writes from multiple processes
问题描述
考虑多个进程打开同一个文件并附加到它的情况.O_APPEND 保证查找到文件末尾然后开始写操作是原子的.因此，多个进程可以附加到同一个文件中，并且只要每个写入大小为 <= PIPE_BUF，任何进程都不会覆盖任何其他进程的写入.
Consider a case where multiple processes open the same file and append to it. O_APPEND guarantees that seeking to the end of file and then beginning the write operation is atomic. So multiple processes can append to the same file and no process will overwrite any other processes' write as far as each write size is <= PIPE_BUF.
我编写了一个测试程序，其中多个进程打开并写入同一个文件 (write(2)).我确保每个写入大小 > PIPE_BUF (4k).我期待看到进程覆盖其他人数据的情况.但这不会发生.我用不同的写入大小进行了测试.这只是运气还是有原因没有发生?我的最终目标是了解附加到同一文件的多个进程是否需要协调它们的写入.
I wrote a test program where multiple processes open and write to the same file (write(2)). I make sure each write size is > PIPE_BUF (4k). I was expecting to see instances where a process overwrites someone else's data. But that doesnt happen. I tested with different write sizes. Is that just luck or is there a reason why that doesn't happen?
My ultimate goal is to understand if multiple processes appending to the same file need to co-ordinate their writes.
这是完整的程序.每个进程创建一个 int 缓冲区，用它的 rank 填充所有值，打开一个文件并写入其中.
Here is the complete program. Every process creates an int buffer, fills all values with its rank, opens a file and writes to it.
规格:OpenMPI 1.4.3 上Opensuse 11.3 64 位
Specs: OpenMPI 1.4.3 on Opensuse 11.3 64-bit
编译为:mpicc -O3 test.c，运行方式: mpirun -np 8 ./a.out
Compiled as: mpicc -O3 test.c, run as: mpirun -np 8 ./a.out
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
int
main(int argc, char** argv) {
int rank, size, i, bufsize = 134217728, fd, status = 0, bytes_written, tmp_bytes_written;
int* buf;
char* filename = "/tmp/testfile.out";
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
buf = (int*) malloc (bufsize * sizeof(int));
if(buf == NULL) {
status = -1;
perror("Could not malloc");
goto finalize;
}
for(i=0; i<bufsize; i++)
buf[i] = rank;
if(-1 == (fd = open(filename, O_APPEND|O_WRONLY, S_IWUSR))) {
perror("Cant open file");
status = -1;
goto end;
exit(-1);
}
bytes_written = 0;
if(bufsize != (tmp_bytes_written = write(fd, buf, bufsize))) {
perror("Error during write");
printf("ret value: %d
", tmp_bytes_written);
status = -1;
goto close;
}
close:
if(-1 == close(fd)) {
perror("Error during close");
status = -1;
}
end:
free(buf);
finalize:
MPI_Finalize();
return status;
}
推荐答案
小于 PIPE_BUF 的写入原子性仅适用于管道和 FIFO.对于文件写入，POSIX 说:
Atomicity of writes less than PIPE_BUF applies only to pipes and FIFOs. For file writes, POSIX says:
本卷 POSIX.1-2008 没有指定并发行为从多个进程写入文件.应用程序应该使用一些并发控制的形式.
This volume of POSIX.1-2008 does not specify behavior of concurrent writes to a file from multiple processes. Applications should use some form of concurrency control.
...这意味着您要靠自己 - 不同的类 UNIX 系统会提供不同的保证.
...which means that you're on your own - different UNIX-likes will give different guarantees.
这篇关于了解来自多个进程的并发文件写入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
