Linux File IO-多线程性能-写入不同的文件 [英] Linux File IO - Multithreading performance - writing to different files

问题描述

我目前正在开发一个音频记录应用程序，该应用程序可以从网络中获取多达8个音频流，并将数据保存到磁盘上(简体;)). 现在，每个流都由一个线程处理->同一线程也在磁盘上进行保存工作.

I'm currently working on an audio recording application, that fetches up to 8 audio streams from the network and saves the data to the disk (simplified ;) ). Right now, each stream gets handled by one thread -> the same thread also does the saving work on the disk.

这意味着我有8个不同的线程在同一张磁盘上执行写操作，每个线程都写到一个不同的文件中.

That means I got 8 different threads that perform writes on the same disk, each one into a different file.

您是否认为，如果所有写入工作都由一个公共线程完成(随后会将数据写入特定文件)，磁盘I/O性能是否会提高?

Do you think there would be an increase in the disk i/o performance if all the writing work would be done by one common thread (that would sequently write the data into the particular files)?

OS是嵌入式Linux，磁盘"是CF卡，应用程序用C编写.

OS is an embedded Linux, the "disk" is a CF card, the application is written in C.

感谢您的想法 尼克

推荐答案

简短的答案:鉴于您正在写入Flash磁盘，因此我不希望线程数会以一种方式或另一种方式产生很大的变化.但是，如果确实有所作为，我希望多个线程比单个线程更快，而不是更慢.

The short answer: Given that you are writing to a Flash disk, I wouldn't expect the number of threads to make much difference one way or another. But if it did make a difference, I would expect multiple threads to be faster than a single thread, not slower.

更长的答案:

我写了一个与您大约6年前描述的程序类似的程序-它运行在嵌入式PowerPC Linux卡上，并向/从SCSI硬盘读取/写入多个同时的音频文件.我最初用一个执行I/O的线程来编写它，因为我认为这样做可以提供最佳的吞吐量，但事实并非如此.

I wrote a similar program to the one you describe about 6 years ago -- it ran on an embedded PowerPC Linux card and read/wrote multiple simultaneous audio files to/from a SCSI hard drive. I originally wrote it with a single thread doing I/O, because I thought that would give the best throughput, but it turned out that that was not the case.

特别是，当多个线程同时读/写时，SCSI层知道来自所有不同线程的所有挂起的请求，并能够对I/O请求进行重新排序，从而可以找到驱动器磁头.最小化.另一方面，在单线程IO方案中，SCSI层仅知道单个下一个"未完成的I/O请求，因此无法进行该优化.在许多情况下，这意味着驱动器磁头需要额外的行程，因此吞吐量会降低.

In particular, when multiple threads were reading/writing at once, the SCSI layer was aware of all the pending requests from all the different threads, and was able to reorder the I/O requests such that seeking of the drive head was minimized. In the single-thread-IO scenario, on the other hand, the SCSI layer knew only about the single "next" outstanding I/O request and thus could not do that optimization. That meant extra travel for the drive head in many cases, and therefore lower throughput.

当然，您的应用程序未使用SCSI或需要寻找磁头的旋转驱动器，因此这对您来说可能不是问题-但是如果有的话，文件系统/硬件层可能会进行其他优化知道多个同时的I​​/O请求.找出问题的唯一真实方法是尝试各种模型并测量结果.

Of course, your application is not using SCSI or a rotating drive with heads that need seeking, so that may not be an issue for you -- but there may be other optimizations that the filesystem/hardware layer can do if it is aware of multiple simultaneous I/O requests. The only real way to find out is to try various models and measure the results.

我的建议是通过将磁盘I/O移到线程池中来将磁盘I/O与网络I/O分离.然后，您可以将I/O线程池的最大大小从1更改为N，并针对每种大小来衡量系统的性能.这样一来，您便可以清楚地了解在特定硬件上最有效的方法，而无需多次重写代码.

My suggestion would be to decouple your disk I/O from your network I/O by moving your disk I/O into a thread-pool. You can then vary the maximum size of your I/O-thread-pool from 1 to N, and for each size measure the performance of the system. That would give you a clear idea of what works best on your particular hardware, without requiring you to rewrite the code more than once.

这篇关于Linux File IO-多线程性能-写入不同的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！