IO性能 - 用于创建和删除高voulme文件 [英] IO performance - for High voulme file creation and deletion
问题描述
我们正在开发一种软件，每小时创建50000个数据文件（大小为3-5 KB）。这些文件将被扫描并保存在数据库中。
超过2天的文件将按计划删除。
I想要知道，这会影响IO性能，因为文件的创建和删除以及如何处理这个问题。
粗略地说，这意味着大约15次写入/秒（与DB相关的I / O没有被考虑在内。通常，这个负载可以由一个体面的和最近的服务器处理硬件配备中高端存储。
一些（有挑战性的:)）评论但是：
- 制作确保你考虑到反病毒所带来的开销和瓶颈。我甚至会考虑不使用任何AV（当然是允许的话）
将这些文件存储在专用分区上（不要弄乱你C：驱动器）- 理想情况下，使用专用存储（控制器，磁盘和hellip;）来防止对其他系统和应用程序进程的任何干扰
- 评估基于SSD的存储，这可能是最适合此类工作负载的，特别是不需要高容量存储的地方
- 如果使用DAS，请选择具有读写缓存的控制器
- 看看这份白皮书，对确定存储配置很有用：
http://www.microsoft.com/ whdc / system / sysperf / Perf_tun_srv2.mspx （以前的Windows版本中存在相同的指导）
- 如果你负担得起，请不要将所有文件放在同一个文件夹中。如果需要手动清理，这将有所帮助
- 谈论应用程序本身的代码，确保正确处置对象和释放句柄（取决于开发语言和技术）并明智地使用内存。除非必要，否则请勿使用事务处理操作。
- 禁用创建8.3名称以及更新"最后修改过的"&lidquo;以及明确地使用多线程
或多进程- 属性可以改善表现。我说'可能'”因为每次使用这种方法时我都没有看到过显着的改进。请参阅
http://technet.microsoft.com/en-us/ library / cc785435（WS.10）.aspx 了解详细信息
- 如果使用2008之前版本的Windows，请评估磁盘分区对齐方式，请参阅
http://msdn.microsoft.com/en-us/library/dd758814（SQL.100）的.aspx 。- 对您的应用程序进行压力测试并捕获与存储相关的计数器
<它不会总是改善，但几乎不会降低
p>注意：尽管ExFAT文件系统可能意味着比NTFS更低的开销，但此时我不建议
We are developing a software that will create 50000 data files (3-5 KB in size) every hour. These file will be scanned and saved in database.
The files older than 2 days will be deleted on a schedule basis.
I want to know, will this affect the IO performance because of creation and deletion of files and how to deal with this.
Roughly, this would mean about 15 writes/second (DB-related I/O’s not taken into account. Normally, this load can be handled by a decent and recent server hardware equipped with mid to high-end storage.
A few (challengeable:) ) comments though:
- Make sure you take into account the overhead and bottleneck an anti-virus would imply. I would even think about not using any AV at all (if allowed of course)
Store those files on a dedicated partition (do not mess you C: drive)- Ideally, use a dedicated storage (controller, disks…) to prevent any interference with other system and application processes
- Evaluate SSD-based storage, which may be the most suitable for this type of workload, particularly where high-volume storage is not a requirement
- If using DAS, chose a controller with both read and write cache
- Have a look at this white paper, which may be useful for determining you storage configuration: http://www.microsoft.com/whdc/system/sysperf/Perf_tun_srv2.mspx (the same guidance exists from previous Windows versions)
- If you can afford it, do not place all files in the same folder. This will help if manual cleanup is needed
- Talking about the code of the application itself, make sure you properly dispose object and release handles (depending on the development language and technique) and wisely use memory. Unless required, do not use transacted operations. Use multi-threading or multi-processes wisely too
- Disabling creation of 8.3 names as well as update of the “Last modified” attribute may improve performances. I say “may” because I have not seen a dramatic improvement each time I use this method. See http://technet.microsoft.com/en-us/library/cc785435(WS.10).aspx for details
- If you use a version of Windows prior 2008, evaluate disk partition alignment, see http://msdn.microsoft.com/en-us/library/dd758814(SQL.100).aspx. It will not always improve but will almost never degrade
- Perform stress tests of your application and capture storage-related counters
Note: Although ExFAT file system may imply a lower overhead than NTFS, I would not recommend it at this point
这篇关于IO性能 - 用于创建和删除高voulme文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
