nodejs加密模块，执行hash.update()将所有输入存储在内存中 [英] nodejs crypto module, does hash.update() store all input in memory

问题描述

我有一个API路由，可以代理从浏览器/客户端到AWS S3的文件上传.

I have an API route that proxies a file upload from the browser/client to AWS S3.

此API路由会尝试在上传文件时流式传输文件，以避免将文件的全部内容缓冲在服务器的内存中.

This API route attempts to stream the file as it is uploaded to avoid buffering the entire contents of the file in memory on the server.

但是，该路由还会尝试计算文件正文的MD5校验和.对文件的每个部分进行分块后，将使用该分块来调用hash.update()方法.

However, the route also attempts to calculate an MD5 checksum of the file's body. As each part of the file is chunked, the hash.update() method is invoked w/ the chunk.

http://nodejs.org/api/crypto.html#crypto_hash_update_data_input_encoding

var crypto = require('crypto');
var hash = crypto.createHash('md5');
function write (chunk) {
  // invoked many times as file is uploaded
  hash.update(chunk);
}
function done() {
  // will hash buffer all chunks in memory at this point?
  hash.digest('hex');
}

Hash实例是否会缓存文件的所有内容以便执行哈希计算(从而违反了避免将整个文件内容缓存在内存中的目标)?还是可以增量计算MD5哈希值，而无需提供全部输入来执行计算?

Will the instance of Hash buffer all the contents of the file in order to perform the hash calculation (thus defeating the goal of avoiding buffering the entire file's contents in memory)? Or can an MD5 hash be calculated incrementally, without ever having the entire input available to perform the calculation?

推荐答案

MD5和其他一些哈希函数基于

MD5 and some other hash functions are based on the Merkle–Damgård construction. It supports the incremental/progressive/streaming hashing of data. After the data is transformed into an internal state (which has a fixed size) a last finalization step is performed to generate the final hash by padding and processing the last block and afterwards by simply returning the final state.

这也可能就是为什么许多哈希库函数以这种方式设计的原因，即具有更新和最终确定步骤.

This is probably also why many hashing library functions are designed in such a way with an update and a finalization step.

要回答您的问题:不，文件内容不会保存在缓冲区中，而是会转换为固定大小的内部状态.

这篇关于nodejs加密模块，执行hash.update()将所有输入存储在内存中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！