Java：高效计算大文件的SHA-256哈希 [英] Java: Calculate SHA-256 hash of large file efficiently
问题描述
我需要计算一个大文件（或其中的一部分）的SHA-256散列。我的实现工作正常，但它比C ++的CryptoPP计算慢得多（对于〜30GB文件，25分钟对10分钟）。我需要的是在C ++和Java中执行相似的执行时间，因此几乎可以同时完成哈希。我也尝试了Bouncy Castle实现，但它给了我相同的结果。下面是我如何计算哈希：
int buff = 16384;
尝试{
RandomAccessFile file = new RandomAccessFile（T：\\someLargeFile.m2v，r）;
long startTime = System.nanoTime（）;
MessageDigest hashSum = MessageDigest.getInstance（SHA-256）;
byte [] buffer = new byte [buff];
byte [] partialHash = null;
long read = 0;
//计算测试的漏洞文件的散列值
long offset = file.length（）;
int unitsize;
while（read
file.read（buffer，0，unitsize）;
hashSum.update（buffer，0，unitsize）;
阅读+ =单位大小;
}
file.close（）;
partialHash =新字节[hashSum.getDigestLength（）];
partialHash = hashSum.digest（）;
long endTime = System.nanoTime（）;
System.out.println（endTime - startTime）;
} catch（FileNotFoundException e）{
e.printStackTrace（）;
}
我的解释可能无法解决您的问题问题，因为它很大程度上取决于您的实际运行时环境，但是当我在我的系统上运行代码时，吞吐量受磁盘I / O限制，而不受哈希计算的限制。切换到NIO并不能解决问题，而只是由于您正在以非常小的部分（16kB）读取文件而造成的。在我的系统增加缓冲区大小（BUFF）到1MB代替16kB的更多的吞吐量了一倍以上，但随着> 50MB / s时，我仍然磁盘速度和不能完全加载的单个CPU核心限制。
BTW：您可以通过围绕一个FileInputStream一个DigestInputStream简化您的实现很多，通过文件读取，并从DigestInputStream计算的散列，而不是从RandomAccessFile的手动洗牌的数据MessageDigest就像你的代码一样。
我用旧的Java版本做了一些性能测试，似乎有一个相关的这里是Java 5和Java 6之间的区别。我不确定SHA实现是否被优化，或者虚拟机是否更快地执行代码。我使用不同的Java版本（1MB缓冲区）获得的吞吐量为：Sun b / s
$ b
- Sun JDK 1.5.0_15（客户端）：28MB / s，受到CPU限制
- Sun JDK 1.5.0_15（服务器）：45MB / s，受CPU限制
- Sun JDK 1.6.0_16（客户端）： 42MB / s，受CPU限制
- Sun JDK 1.6.0_16（服务器）：52MB / s，受磁盘I / O限制（85-90％CPU负载） b
$ b我对CryptoPP SHA实现中汇编器部分的影响有点好奇，因为基准测试结果表明SHA-256算法只需要Opteron 15.8个CPU周期/字节。我很遗憾没能与cygwin的（构建成功，但生成的exe立即失败）海湾合作委员会建立CryptoPP，但并没有CryptoPP汇编支持建设有VS2005（默认发布配置）性能基准测试并比较了Java SHA在内存缓冲区中实现，省去了任何磁盘I / O，我在2.5GHz Phenom上得到了以下结果：
- Sun JDK1.6.0_13（服务器）：26.2个周期/字节
- CryptoPP（C ++只）：21.8个周期/字节
- CryptoPP（汇编）：13.3周期/字节
这两个基准计算4GB空字节数组的SHA散列，遍历它在1MB的块，这是通过MessageDigest＃update（Java）或CryptoPP的SHA256.Update函数（C ++）。
我能够在运行Linux的虚拟机中使用gcc 4.4.1（-O3）构建并测试CryptoPP，并且只获得了appr。与VS exe的结果相比吞吐量的一半。我不确定这个虚拟机有多大的差异，以及VS通常会产生比gcc更好的代码，但是我现在无法从gcc获得更准确的结果。
I need to calculate a SHA-256 hash of a large file (or portion of it). My implementation works fine, but its much slower than the C++'s CryptoPP calculation (25 Min. vs. 10 Min for ~30GB file). What I need is a similar execution time in C++ and Java, so the hashes are ready at almost the same time. I also tried the Bouncy Castle implementation, but it gave me the same result. Here is how I calculate the hash:
int buff = 16384; try { RandomAccessFile file = new RandomAccessFile("T:\\someLargeFile.m2v", "r"); long startTime = System.nanoTime(); MessageDigest hashSum = MessageDigest.getInstance("SHA-256"); byte[] buffer = new byte[buff]; byte[] partialHash = null; long read = 0; // calculate the hash of the hole file for the test long offset = file.length(); int unitsize; while (read < offset) { unitsize = (int) (((offset - read) >= buff) ? buff : (offset - read)); file.read(buffer, 0, unitsize); hashSum.update(buffer, 0, unitsize); read += unitsize; } file.close(); partialHash = new byte[hashSum.getDigestLength()]; partialHash = hashSum.digest(); long endTime = System.nanoTime(); System.out.println(endTime - startTime); } catch (FileNotFoundException e) { e.printStackTrace(); }解决方案My explanation may not solve your problem since it depends a lot on your actual runtime environment, but when I run your code on my system, the throughput is limited by disk I/O and not the hash calculation. The problem is not solved by switching to NIO, but is simply caused by the fact that you're reading the file in very small pieces (16kB). Increasing the buffer size (buff) on my system to 1MB instead of 16kB more than doubles the throughput, but with >50MB/s, I am still limited by disk speed and not able to fully load a single CPU core.
BTW: You can simplify your implementation a lot by wrapping a DigestInputStream around a FileInputStream, read through the file and get the calculated hash from the DigestInputStream instead of manually shuffling the data from a RandomAccessFile to the MessageDigest as in your code.
I did a few performance tests with older Java versions and there seem to be a relevant difference between Java 5 and Java 6 here. I'm not sure though if the SHA implementation is optimized or if the VM is executing the code much faster. The throughputs I get with the different Java versions (1MB buffer) are:
- Sun JDK 1.5.0_15 (client): 28MB/s, limited by CPU
- Sun JDK 1.5.0_15 (server): 45MB/s, limited by CPU
- Sun JDK 1.6.0_16 (client): 42MB/s, limited by CPU
- Sun JDK 1.6.0_16 (server): 52MB/s, limited by disk I/O (85-90% CPU load)
I was a little bit curious on the impact of the assembler part in the CryptoPP SHA implementation, as the benchmarks results indicate that the SHA-256 algorithm only requires 15.8 CPU cycles/byte on an Opteron. I was unfortunately not able to build CryptoPP with gcc on cygwin (the build succeeded, but the generated exe failed immediately), but building a performance benchmark with VS2005 (default release configuration) with and without assembler support in CryptoPP and comparing to the Java SHA implementation on an in-memory buffer, leaving out any disk I/O, I get the following results on a 2.5GHz Phenom:
- Sun JDK1.6.0_13 (server): 26.2 cycles/byte
- CryptoPP (C++ only): 21.8 cycles/byte
- CryptoPP (assembler): 13.3 cycles/byte
Both benchmarks compute the SHA hash of a 4GB empty byte array, iterating over it in chunks of 1MB, which are passed to MessageDigest#update (Java) or CryptoPP's SHA256.Update function (C++).
I was able to build and benchmark CryptoPP with gcc 4.4.1 (-O3) in a virtual machine running Linux and got only appr. half the throughput compared to the results from the VS exe. I am not sure how much of the difference is contributed to the virtual machine and how much is caused by VS usually producing better code than gcc, but I have no way to get any more exact results from gcc right now.
这篇关于Java：高效计算大文件的SHA-256哈希的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
