如何生成校验和在不溢出RAM的情况下将非常大的文件转换为Javascript中的64位? [英] How to generate checksum & convert to 64 bit in Javascript for very large files without overflowing RAM?
问题描述
问题:
-
如何正确生成校验和,这是唯一的,一致的,独立的浏览器?另外,我想将SHA256 / MD5校验和字符串转换为64位。
How to generate a checksum correctly, which is unique, consistent independent of browsers? Also, I would like to convert a SHA256/MD5 checksum string to 64-bit.
如何在不占用大量RAM的情况下正确读取文件以生成校验和?即我们如何处理1 GB的文件而不会损害RAM
How to properly read a file without huge RAM requirement to generate checksum? i.e. how do we deal with 1 GB file without compromising RAM
例如是否可以在不将文件加载到内存的情况下读取文件?(请参见答案)
此项目看起来很有希望,但也无法使它起作用
This project seems promising, but couldn't get it worked either.
我的意图是逐步/递增地在X MB块中生成校验和。这可能有助于避免一次使用过多的RAM。
以下代码无法正常工作:
My intention is to generate the checksum progressively/incrementally in chunks of X MBs. This may help to avoid using too much RAM at a time.
Following is the code, which is not working as expected:
let SIZE_CHECKSUM = 10 * Math.pow(1024, 2); // 10 MB; But can be 1 MB too
async function GetChecksum (file: File):
Promise<string>
{
let hashAlgorithm: CryptoJS.lib.IHasher<Object> = CryptoJS.algo.SHA256.create();
let totalChunks: number = Math.ceil(file.size / SIZE_CHECKSUM);
for (let chunkCount = 0, start = 0, end = 0; chunkCount < totalChunks; ++chunkCount)
{
end = Math.min(start + SIZE_CHECKSUM, file.size);
let resultChunk: string = await (new Response(file.slice(start, end)).text());
hashAlgorithm.update(resultChunk);
start = chunkCount * SIZE_CHECKSUM;
}
let long: bigInt.BigInteger = bigInt.fromArray(hashAlgorithm.finalize().words, 16, false);
if(long.compareTo(bigInt.zero) < 0)
long = long.add(bigInt.one.shiftLeft(64));
return long.toString();
}
在不同的浏览器中显示不同的结果。
It shows different results in different browsers.
推荐答案
下面的代码行中有一个逻辑问题:
There is a logical issue in the code at below line:
start = chunkCount * SIZE_CHECKSUM; // <--- bug
变量 start
初始化为0,然后在第一次迭代中再次重置为0 ,这是不对的。
以下是使用以下方法获取32字节SHA5校验和的方法:问题中提到的同一库: emn178 / js-sha256 。
The variable start
is initialized to 0 and then again reset to 0 in the 1st iteration, which is not right.
Following is the way to get a 32 bytes SHA5 checksum with the same library mentioned in the question: "emn178/js-sha256".
该库不提供Typescript接口,但是我们可以定义如下:
That library doesn't provide a Typescript interface, but we can define trivially as following:
// Sha256.d.ts (also name the corresponding JS file as "Sha256.js")
declare class Sha256 {
update (data: ArrayBuffer): Sha256;
hex (): string;
}
declare var sha256: any;
declare interface sha256 {
create (): Sha256;
}
然后按以下方式使用它:
Then use it as following:
import "./external/Sha256"
async function GetChecksum (file: File):
Promise<string>
{
let algorithm = sha256.create();
for(let chunkCount = 0, totalChunks = Math.ceil(file.size / SIZE_CHECKSUM);
chunkCount < totalChunks;
++chunkCount)
{
let start = chunkCount * SIZE_CHECKSUM, end = Math.min(start + SIZE_CHECKSUM, file.size);
algorithm.update(await (new Response(file.slice(start, end)).arrayBuffer()));
}
return algorithm.hex();
}
以上代码在我的所有浏览器中针对任何块大小生成相同的校验和。
Above code generates same checksums in all my browsers for any chunk size.
这篇关于如何生成校验和在不溢出RAM的情况下将非常大的文件转换为Javascript中的64位?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!