在异步写入文件时如何锁定文件 [英] How can I lock a file while writing to it asynchronously

问题描述

我有两个正在运行的节点线程，一个正在监视目录中的文件使用情况，另一个负责将文件写入给定目录.

I've got two node threads running, one watching a directory for file consumption and another that is responsible for writing files to given directories.

通常，它们不会在同一目录上运行，但是对于我正在使用的极少数情况，它们将是相同的.

Typically they won't be operating on the same directory, but for an edge case I'm working on they will be.

看来，使用该应用的应用程序在完全写入文件之前就已对其进行抓取，从而导致文件损坏.

It appears that the consuming app is grabbing the files before they are fully written, resulting in corrupt files.

有没有一种方法可以锁定文件，直到写入完成?我已经研究过lockfile模块，但是不幸的是，我认为它不适用于此特定应用程序.

Is there a way I can lock the file until the writing is complete? I've looked into the lockfile module, but unfortunately I don't believe it will work for this particular application.

=====

完整的代码远远超出了这里的意义，但要点是:

The full code is far more than makes sense to put here, but the gist of it is this:

1. 应用程序剥离了观察者和听众

监听器:

• 侦听要添加到数据库中的文件，然后使用fs.writeFile
将其导出

• listen for file being added to db, export it using fs.writeFile

观察者:

• watcher使用chokidar跟踪每个受监视目录中的已添加文件
• 当找到fs.access时将被调用以确保我们可以访问该文件
  • fs.access似乎并没有因为正在写入文件而感到困惑
  • watcher uses chokidar to track added files in each watched directory
  • when found fs.access is called to ensure we have access to the file
    • fs.access seems to be unfazed by file being written
    • 文件流是必需的，因为我们需要文件哈希

    在这种情况下，文件被导出到监视目录，然后重新导入 通过观看过程.

    In this case the file is exported to the watched directory and then reimported by the watch process.

    推荐答案

    编写锁定状态系统实际上非常简单.我找不到我在哪里做的，但想法是:

    Writing a lock-state system is actually pretty simple. I can't find where I did this, but the idea is to:

    1. 每当您获得锁时创建锁文件，
    2. 释放锁时将其删除，
    3. 在发生超时后删除它们，
    4. 每当请求一个已存在锁定文件的文件的锁定时，都将抛出该异常.

    锁定文件只是单个目录中的空文件.每个锁定文件从其表示的文件的完整路径的哈希值中获得名称.我使用了MD5(速度相对较慢)，但是只要您确信路径字符串不会发生冲突，任何哈希算法都应该可以.

    A lock file is simply an empty file in a single directory. Each lock file gets its name from the hash of the full path of the file it represents. I used MD5 (which is relatively slow), but any hashing algo should be fine as long as you are confident there will be no collisions for path strings.

    这不是100％线程安全的，因为(除非我错过了一些愚蠢的东西)您无法原子检查文件是否存在并在Node中创建它，但是在我的用例中，我持有用于10秒或更长时间，因此微秒级的比赛条件似乎并没有太大的威胁.如果您每秒在同一文件上持有并释放数千个锁，那么这种竞争状况可能适用于您.

    This isn't 100% thread-safe, since (unless I've missed something stupid) you can't atomically check if a file exists and create it in Node, but in my use case, I was holding locks for 10 seconds or more, so microsecond race conditions didn't seem that much of a threat. If you are holding and releasing thousands of locks per second on the same files, then this race condition might apply to you.

    很明显，这些将仅是建议性锁定，因此您可以确保代码请求锁定并捕获预期的异常.

    These will be advisory locks only, clearly, so it is up to you to ensure your code requests locks and catches the expected exceptions.

    这篇关于在异步写入文件时如何锁定文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！