md5增量哈希一个大文件吗? [英] md5 hash a large file incrementally?

问题描述

在浏览器中，我使用JS FileReader().readAsBinaryString()读取了一个文件.使用CryptoJS库，我可以MD5哈希数据.

In the browser, I read in a file using the JS FileReader().readAsBinaryString(). Using the CryptoJS library I can MD5 hash the data.

这很好，但是我不知道如何处理大文件.例如.仅读取2GiB文件会使浏览器窗口崩溃.我可以随时从文件数据中切出斑点并进行哈希处理，但这不会阻止其他人在不遵循与我相同的步骤的情况下验证相同的哈希吗?

This works fine but I do not know how to handle large files. E.g. Just reading a 2GiB file crashes the browser window. I can slice blobs from the file data and hash that as I go but wouldn't this prevent anyone else from verifying the same hash without following the same steps as me?

在这种情况下，是否有办法获取大文件的md5哈希?例如，您将如何计算1TB文件的md5哈希值?我需要以流形式读取文件吗?

Is there a way to get the md5 hash of a large file in this circumstance? How would you calc the md5 hash of a 1TB file, for example? Do I need to read the file in as a stream?

这是我第一次咬牙，我不确定该怎么做.

First time cutting my teeth on this one and I'm not sure how to do it.

这驻留在有角度的指令中，因此位于范围内.

This resides in an angular directive, hence the scope.

var reader = new FileReader();
                reader.onload = function (loadEvent) {
                    scope.$apply(function () {
                        scope.files = changeEvent.target.files;
                        scope.fileread = loadEvent.target.result;
                        scope.md5Data = CryptoJS.MD5(scope.fileread).toString();
                    });
                }
                // First ten megs of the file
                reader.readAsBinaryString((changeEvent.target.files[0]).slice(0, 10 * 1024 * 1024));

推荐答案

我可以随时从文件数据中切出斑点并进行哈希处理，但这不会阻止其他人在不遵循与我相同的步骤的情况下验证相同的哈希吗?

I can slice blobs from the file data and hash that as I go but wouldn't this prevent anyone else from verifying the same hash without following the same steps as me?

是的，因此，这正是MD5算法在其合同中提供的内容:

Yes, therefore this is exactly what the MD5 algorithm provides in its contract:

1. 您有一个文件
2. 通过添加单个"1"和多个"0"来填充文件，因此该文件可被512整除.
3. 每个回合都会计算文件512字节的一个切片的md5哈希值，并将其与先前的结果组合.

因此您无需重复这些步骤，并确保其他用户也可以这样做.

So you will not need to repeat these steps and make sure another user does the same.

由于MD5是按块计算的，因此可以进行流传输，您可以在此处阅读(尽管使用了nodejs的crypt模块，它是googlecode项目crypto-js的模块化端口.):

Since MD5 is computed in blocks, streaming is possible, as you can read here (although done with the crypt module of nodejs which is a modularized port of googlecode project crypto-js.):

http://www.hacksparrow.com/how-to-generate-md5-sha1-sha512-sha256-checksum-hashes-in-node-js.html

这篇关于md5增量哈希一个大文件吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！