比较s3对象和本地对象的准确方法 [英] accurate method to compare the s3 object and local object

问题描述

我在s3中将修改后的时间（以毫秒为单位）和文件大小保存到了对象的元数据中。我意识到即使打开文件也没有更改任何内容，然后保存文件而不进行编辑。修改的时间将被更改，在这种情况下，它将更新s3对象。我曾想过要使用尺寸，但是尺寸也不会那么准确，因为即使修改后尺寸也有可能保持不变。我还使用了 Binary 从 s3.getObject 和本地文件 Binary ，但没有任何更改。 Binary 也不会相同。

I saved the modified time in ms and file size into the object's metadata in s3. I realized even if I did not change anything for my file if I open it then just save the file without editing. The modified time will be changed and in this case it will update the s3 object. I thought of using size but size wouldn't be as accurate too because there are chances for the size to be the same even after modified. I also used the Binary got back from s3.getObject and locally's file Binary but without any changes. The Binary wont be the same too. What would be a better more accurate way to track changes?

我的代码中有这样的东西，可以保存修改后的ms和文件大小

I have something like this in my code which saves the file modified ms and file size

fs.readFile(path, async (err, fileBinary) => {
      if (err) throw err;

      const s3 = new AWS.S3();
      const Key = path.replace(process.env.WATCH_PATH, '');
      const filename = Key.split('/').pop();

      // if filename is within the regex, ignore the file.  Do nothing.
      if (new RegExp(IGNORE_FILES_TO_S3()).test(filename)) return false;

      const getStat = await getFileStat(path);
      // console.log(getStat, 'getstatsssssssssssssss');
      const s3PutParams = {
          Body: fileBinary,
          Bucket: process.env.S3_BUCKET,
          Key,
          Metadata: {  // thought of saving these two as comparison in future usage, which works but really really accurate though
              mtimeMs: String(getStat.mtimeMs),
              size: String(getStat.size)
          }
      };
// rest of the code here just do comparisons and decide if `s3.putOjbect` should be done or not.
});

我的 getFileStat（）

exports.getFileStat = (path) => {
    /*
    SAMPLE: success
    {
    dev: 2097,
    mode: 33204,
    nlink: 1,
    uid: 1000,
    gid: 1000,
    rdev: 0,
    blksize: 4096,
    ino: 5639856,
    size: 2,
    blocks: 8,
    atimeMs: 1545952029779.866,
    mtimeMs: 1545952020431.9802,
    ctimeMs: 1545952020439.98,
    birthtimeMs: 1545952020439.98,
    atime: 2018-12-27T23:07:09.780Z,
    mtime: 2018-12-27T23:07:00.432Z,
    ctime: 2018-12-27T23:07:00.440Z,
    birthtime: 2018-12-27T23:07:00.440Z
    }
    */
    return new Promise((res, rej) => {
        fs.stat(path, (err, stat) => {
            if (err) rej(err);
            res(stat);
        });
    });
};

在此先感谢您的建议和帮助。

Thanks in advance for any suggestions and help.

PS。这并不是将任何内容保存到数据库中，因此，如果有将内容保存到数据库中以进行比较的想法，则根本不会保存任何信息

PS. This is not saving anything into the DB so no info will be saved at all in case there's an idea of saving something into DB for comparison purpose

推荐答案

要将本地文件的内容与Amazon S3对象进行比较，请使用 ETag ，这是对内容的校验和。检索有关S3对象的信息时，可以使用ETag。

To compare the contents of a local file with an Amazon S3 object, use the ETag, which is a checksum on the contents. The ETag is available when retrieving information about an S3 object.

请参阅：有关AWS S3 ETag的所有信息-Teppen.io

此外，请注意，通过分段上传上传的对象具有稍微复杂一点的计算。请参阅：对于大于5GB的文件，如何计算Amazon-S3 Etag的算法是什么？

Also, please note that an object uploaded via multi-part upload has a slightly more complex calculation. See: What is the algorithm to compute the Amazon-S3 Etag for a file larger than 5GB?

这篇关于比较s3对象和本地对象的准确方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！