使用MappedByteBuffer在普通的FileChannel上顺序编写长文件是否有性能优势？ [英] Is there a performance advantage in writing a long file sequentially using MappedByteBuffer over a plain FileChannel?

问题描述

我想了解调用FileChannel.write与16k缓冲区的短暂连续和AND映射多个ByteBuffers与16k的附加大小之间的差异，这里描述： https://stackoverflow.com/a/7367952/962872

我认为映射的bytebuffer方法产生在每次追加后放弃MappedByteBuffers会造成很多垃圾。而且我也不确定它是否更快。你还需要做一堆映射操作...（每个追加一个）。

或者你应该映射一个巨大的ByteBuffer（尽可能大）和只是继续写这个MappedByteBuffer吗？

我正在使用FileChannel.write方法与一个Java端16kb缓冲区作为一个快速的方式来写一个文件，但我想以确保我不会错过任何东西更快/更好。

任何人都可以点亮？

解决

我认为映射的bytebuffer方法会产生大量垃圾，因为您在每次追加后放弃MappedByteBuffers。 b
$ b

如果您创建了16 KB的缓冲区，如果你创建了1 GB的缓冲区，就不需要那么多了，这样做的代价很小（假设你有一个64位的JVM）

lockquote

我正在使用FileChannel.write方法和一个Java端的16kb缓冲区作为一个快速的方式来编写一个文件，但是我想确保我不会错过更快/更好的东西。

我会检查你是不是已经写入速度快或者比你开车写的速度还要快。内存映射文件给你的主要优势是低得多的典型延迟。您可以写入的吞吐量将受到驱动器速度的限制。如果你有一个典型的固态硬盘，你的CPU仍然可以写出比它消耗的速度更快的数据，而且如果你使用硬盘驱动器，它可能并不重要，因为硬盘速度太慢了。

典型写入吞吐量

• 现代CPU：2,000 - 6,000 MB / sec
  一个SSD：300-1,200MB / sec
  一个磁盘控制器：200-600MB / sec
  一个HDD：20-60 MB / sec。
  

通常，当CPU不是瓶颈时，你在软件中做什么对应用程序的性能几乎没有影响。

在延迟方面，典型的延迟是

• FileChannel.write ）：10 - 40微秒，但可以超过1毫秒。
  
• 写入内存映射文件：对于短消息，为100纳秒，但可以在达到100毫秒时激活一个新的内存映射是必需的。

I am trying to understand what would be the difference between calling FileChannel.write in short successions with a 16k buffer AND mapping multiple ByteBuffers with an append size of 16k as described here: https://stackoverflow.com/a/7367952/962872

I would think that the mapped bytebuffer approach produces a lot of garbage as you discard the MappedByteBuffers after each append. And I am not sure it is faster either. And you still have to do a bunch of mapping operations... (one per append).

Or perhaps you should map a huge ByteBuffer (as big as possible) and just keep writing to this MappedByteBuffer?

I am using the FileChannel.write approach with a Java-side 16kb buffer as a "fast" way to write a file but I want to make sure I am not missing something faster/better.

Can anyone shed a light?

解决方案

I would think that the mapped bytebuffer approach produces a lot of garbage as you discard the MappedByteBuffers after each append.

It would if you created 16 KB buffers. If you created 1 GB buffers would won't need as many and there is little penalty for doing so (assuming you have a 64-bit JVM)

I am using the FileChannel.write approach with a Java-side 16kb buffer as a "fast" way to write a file but I want to make sure I am not missing something faster/better.

I would check you are not already writing as fast or faster than you drive can write anyway. The main advantage Memory Mapped Files give you is much lower typical latency. The throughput you can write will be limited by the speed of your drive. If you have a typical SSD, your CPU can still write out data faster than it can consume and if you use HDD it probably doesn't matter what you do as the drive is so much slower.

Typical write throughput of

• a modern CPU: 2,000 - 6,000 MB/sec
• an SSD: 300 - 1,200 MB/sec
• a disk controller: 200 - 600 MB/sec
• an HDD: 20 -60 MB/sec.

Often when CPU is not the bottleneck, what you do in software has little impact on the performance of the application.

In terms of latency, the typical latency of

• FileChannel.write() : 10 - 40 micro-seconds but can spike to over one millisecond.
• Writing to memory mapped file : 100 nano-second for a short message but can spike to 100 mi-seconds when a new memory mapping is required.

这篇关于使用MappedByteBuffer在普通的FileChannel上顺序编写长文件是否有性能优势？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！