跨平台和跨进程原子诠释写入文件 [英] Cross-platform and cross-process atomic int writes on file

问题描述

我正在编写一个应用程序，它必须能够处理对它的许多并发访问，不管是按线程还是按进程。所以不应该使用互斥锁或锁。

为了使锁的使用降到最低，我设计的文件是所以所有的数据首先被附加到磁盘上，然后指向它已经更新的信息的地址被改变以引用新的数据。所以我将需要实现一个小的锁定系统，只是为了改变这个int，所以它引用了新的地址。
最好的办法是怎么做的？

我正在考虑在地址前加上一个标记，当它被设置时，读者将会使用旋转锁定，直到释放。但是，恐怕这根本不是原子，是吗？
eg

• 阅读器读取标志，并且未设置
  
• 同一时间，作家写入标志并更改int的值
  
• 读者可能会读取不一致的值！

我正在寻找锁定技术，但是我发现所有的都是用于线程锁定技术，或者锁定整个文件，而不是字段。这是不可能的吗？如何仅追加数据库处理这个？

编辑：
我正在看如何附加数据库（couchDB）做到这一点，似乎他们使用一个线程只将序列化写入文件。这是否意味着它不可能使它们像sqlite一样可嵌入，而不锁定文件系统锁定的整个文件？

谢谢！
Cauê

解决方案

注意文件系统的附加语义 - 它可能不提供原子追加操作。一个选项是把你的文件的内存映射（ mmap ）作为共享，然后执行原子内存操作，比如compare-并在指针上交换。你的成功将取决于你的操作系统是否有这样的操作（Linux，OSX）。

一个正确的（虽然我不确定它是快速的）方式完成你想要的是 rename - 这是大多数文件系统上的原子文件操作。将最新的数据保存在官方文件位置。要更新数据，请将新数据写入临时文件，然后将该临时文件重命名为官方位置。

I'm writing an application that will have to be able to handle many concurrent accesses to it, either by threads as by processes. So no mutex'es or locks should be applied to this.

To make the use of locks go down to a minimum, I'm designing for the file to be "append-only", so all data is first appended to disk, and then the address pointing to the info it has updated, is changed to refer to the new one. So I will need to implement a small lock system only to change this one int so it refers to the new address. How is the best way to do it?

I was thinking about maybe putting a flag before the address, that when it's set, the readers will use a spin lock until it's released. But I'm afraid that it isn't at all atomic, is it? e.g.

• a reader reads the flag, and it is unset
• on the same time, a writer writes the flag and changes the value of the int
• the reader may read an inconsistent value!

I'm looking for locking techniques but all I find is either for thread locking techniques, or to lock an entire file, not fields. Is it not possible to do this? How do append-only databases handle this?

edit: I was looking at how append-only db's (couchDB) do it, and it seems they use a thread only to serialize the writes to file. Does that mean it isn't possible to make them embeddable, like sqlite, without locking the entire file with file system locks?

Thanks! Cauê

解决方案

Be careful about the append semantics of your filesystem - it probably doesn't provide atomic append operations.

One option is to memory map (mmap) your file as shared, then do atomic memory operations like compare-and-swap on the pointer. Your success will depend on whether your OS has such an operation (Linux, OSX do).

A correct (although I'm not sure it is fast) way accomplish what you want is with rename - it is an atomic file operation on most filesystems. Keep the most up-to-date data in an official file location. To update the data, write your new data to a temporary file, then rename that temporary file to the official location.

这篇关于跨平台和跨进程原子诠释写入文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！