具有流的文件I / O - 最佳内存缓冲区大小 [英] File I/O with streams - best memory buffer size
问题描述
我正在写一个小型I / O库来协助一个较大的（爱好）项目。该库的一部分对文件执行各种功能，该文件通过 FileStream 对象进行读取/写入。在每个 StreamReader.Read（...） pass，
我发射一个将用于主要应用程序显示进度信息。循环中进行的处理被维护，但并不耗费时间（例如，它可能是一个简单的文件副本，或者可能涉及加密...）。
$ b $我的主要问题是：使用最好的内存缓冲区大小是多少？考虑到物理磁盘布局，我可以选择2k，这将覆盖CD扇区大小，是512字节硬盘扇区的一个很好的倍数。更高的抽象树，你可以去一个更大的缓冲区，可以一次读整个FAT集群。我意识到与今天的电脑，我可以去一个更多的内存饥饿选项（例如几个MiB），但是然后我增加了UI更新之间的时间，用户感觉到一个响应较差的应用程序。 $ b $除此之外，我最终希望能够提供与FTP / HTTP服务器（通过本地网络/快速DSL）托管的文件的类似接口。那么那些最好的内存缓冲区大小（再一次是感知响应性与性能之间的最佳情况权衡）？
文件已经被文件系统缓存缓冲。您只需要选择一个缓冲区大小，不会强制FileStream使本机Windows ReadFile（）API调用太频繁地填充缓冲区。不要低于千字节，超过16 KB是浪费记忆，不友好的CPU的 L1缓存（通常为16或32 KB的数据）。
4 KB是传统的选择，即使这将完全跨越虚拟内存只有偶然的页面。很难描述;您将最终测量读取缓存文件所需的时间。如果数据在缓存中可用，则以RAM速度运行5 GB /秒。它将在缓存中第二次运行测试，这不会在生产环境中经常发生。文件I / O完全由磁盘驱动器或 NIC 主导，并且速度很慢，复制数据是花生4 KB可以正常运行。
I am writing a small I/O library to assist with a larger (hobby) project. A part of this library performs various functions on a file, which is read / written via the FileStream object. On each StreamReader.Read(...) pass,
I fire off an event which will be used in the main app to display progress information. The processing that goes on in the loop is vaired, but is not too time consuming (it could just be a simple file copy, for example, or may involve encryption...).
My main question is: What is the best memory buffer size to use? Thinking about physical disk layouts, I could pick 2k, which would cover a CD sector size and is a nice multiple of a 512 bytes hard disk sector. Higher up the abstraction tree, you could go for a larger buffer which could read an entire FAT cluster at a time. I realise with today's PC's, I could go for a more memory hungry option (a couple of MiB, for example), but then I increase the time between UI updates and the user perceives a less responsive application.
As an aside, I'm eventually hoping to provide a similar interface to files hosted on FTP / HTTP servers (over a local network / fastish DSL). What would be the best memory buffer size for those (again, a "best-case" tradeoff between perceived responsiveness vs. performance)?
Files are already buffered by the file system cache. You just need to pick a buffer size that doesn't force FileStream to make the native Windows ReadFile() API call to fill the buffer too often. Don't go below a kilobyte, more than 16 KB is a waste of memory and unfriendly to the CPU's L1 cache (typically 16 or 32 KB of data).
4 KB is a traditional choice, even though that will exactly span a virtual memory page only ever by accident. It is difficult to profile; you'll end up measuring how long it takes to read a cached file. Which runs at RAM speeds, 5 gigabytes/sec and up if the data is available in the cache. It will be in the cache the second time you run your test, and that won't happen in a production environment too often. File I/O is completely dominated by the disk drive or the NIC and is glacially slow, copying the data is peanuts. 4 KB will work fine.
这篇关于具有流的文件I / O - 最佳内存缓冲区大小的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
