两个进程写入一个文件，防止混合输出 [英] two processes write to one file, prevent mixing the output

问题描述

我想从两个进程中获取输出并将它们合并到一个文件中，例如:

I want to get output from two processes and merge them into one file, like:

proc1 >> output &
proc2 >> output &

问题在于输出可能会混入最终文件中. 例如，如果第一个进程写道:

The problem is that output may be mixed up in the final file. For example if first process writes:

hellow

hellow

第二个过程写道:

再见

结果可能类似于:

hebylloe

hebylloe

但是我希望它们像(顺序并不重要)这样在单独的行中:

but I expect them to be in seperate lines like (order is not important):

再见

你好

因此，我使用了flock通过以下脚本来同步写入文件:

So I used flock to synchronize writing to the file with the following script:

exec 200>>output
while read line;
  flock -w 2 200
  do echo $line>>output
  flock -u 200
done

并运行以下过程:

proc1 | script &
proc2 | script &

现在的问题是性能显着下降.没有同步，每个进程可以以4MB/秒的速度写入，但是使用同步脚本，写入速度为1MB/秒.

Now the problem is that the performance is decreased significantly. without synchronization each process could write with the speed of 4MB/sec but using the synchronization script the write speed is 1MB/sec.

有人可以帮助我如何合并两个流程的输出并防止混淆输出吗?

Can anyone help me how to merge the output from two processes and prevent mixing outputs up?

我意识到行长和标准缓冲区大小之间存在关系，如果每行的大小小于标准缓冲区大小，那么一切正常，至少没有什么混杂(至少在我的测试中).所以我用bufsize命令运行了每个脚本:

edit: I realized that there is a relation between line length and std buffer size, if size of each line is less than std buffer size, then every thing works well, nothing is mixed (at least in my tests). so I ran each script with bufsize command:

bufsize -o10KB proc1 | script &
bufsize -o10KB proc2 | script &

现在，我要确保此解决方案是防弹的.我找不到缓冲区大小和现在发生的事情之间的任何关系！！！

Now I want to make sure that this solution is bulletproof. I can not find any relation between buffer size and what happens now!!!

推荐答案

现在，我要确保此解决方案是防弹的.我不能 找到缓冲区大小和现在发生的事情之间的任何关系！

Now I want to make sure that this solution is bulletproof. I can not find any relation between buffer size and what happens now!!!

对于完全缓冲的输出流，缓冲区大小决定使用单个写入的数据量. write(2) 调用.对于行缓冲的输出流，使用单个 write(2) 调用编写一行只要不超过缓冲区大小即可.

For a fully buffered output stream, the buffer size determines the amount of data written with a single write(2) call. For a line buffered output stream, a line is written with a single write(2) call as long as it doesn't exceed the buffer size.

如果文件使用 打开(2) O_APPEND ，文件偏移量首先设置为文件末尾 在写之前.文件偏移量的调整和写入 操作是原子步骤.

If the file was open(2)ed with O_APPEND, the file offset is first set to the end of the file before writing. The adjustment of the file offset and the write operation are performed as an atomic step.

另请参阅以下答案:

• write(2)对本地文件系统的原子性
• 了解来自多个进程的并发文件写入

• Atomicity of write(2) to a local filesystem
• Understanding concurrent file writes from multiple processes

这篇关于两个进程写入一个文件，防止混合输出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！