Ubuntu Linux中的异步IO io_submit延迟 [英] asynchronous IO io_submit latency in Ubuntu Linux

查看：148
本文介绍了Ubuntu Linux中的异步IO io_submit延迟的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我正在寻找有关如何为在Ubuntu Linux 14.04上运行的应用程序获取高效和高性能异步IO的建议.

I am looking for advice on how to get efficient and high performance asynchronous IO working for my application that runs on Ubuntu Linux 14.04.

我的应用处理交易并在磁盘/闪存上创建文件.随着应用程序在交易过程中的进行，将创建其他块，这些块必须附加到磁盘/闪存上的文件中.该应用程序在处理新交易时，还需要经常读取该文件的块.每个事务可能还需要从该文件读取一个不同的块，此外还必须创建一个必须附加到该文件的新块.事务进入队列，应用程序可以继续处理队列中的事务，以创建足够深的IO操作管道，以隐藏磁盘或闪存上的读取访问或写入完成的延迟.对于尚未被写入磁盘/闪存的读取块(先前的事务已将其放入写入队列中)，应用程序将停止运行，直到完成相应的写入为止.

My app processes transactions and creates a file on disk/flash. As the app is progressing through transactions additional blocks are created that must be appended to the file on disk/flash. The app needs also to frequently read blocks of this file as it is processing new transactions. Each transaction might need to read a different block from this file in addition to also creating a new block that has to be appended to this file. There is an incoming queue of transactions and the app can continue to process transactions from the queue to create a deep enough pipeline of IO ops to hide the latency of read accesses or write completions on disk or flash. For a read of a block (which was put in the write queue by a previous transaction) that has not yet been written to disk/flash, the app will stall until the corresponding write completes.

我有一个重要的性能目标–应用程序应尽可能降低发布IO操作的延迟.我的应用大约需要10微秒来处理每个事务，并准备对磁盘/闪存上的文件进行写入或读取操作.发出异步读取或写入的额外延迟应尽可能小，以便在仅需要文件写入时，应用程序可以以接近每个事务10个usecs的速率完成每个事务的处理.

I have an important performance objective – the app should incur the lowest possible latency to issue the IO operation. My app takes approximately 10 microseconds to process each transaction and be ready to issue a write to or a read from the file on disk/flash. The additional latency to issue an asynchronous read or write should be as small as possible so that the app can complete processing each transaction at a rate as close to 10 usecs per transaction as possible, when only a file write is needed.

我们正在尝试使用io_submit发出写入和读取请求的实现.如果您对我们所要求的最佳方法有任何建议或反馈，我将不胜感激. io_submit是否可以为我们提供最佳性能以满足我们的目标?对于每个写入io_submit的延迟和每个读取io_submit的延迟，我应该期待什么?

We are experimenting with an implementation that uses io_submit to issue write and read requests. I would appreciate any suggestions or feedback on the best approach for our requirement. Is io_submit going to give us the best performance to meet our objective? What should I expect for the latency of each write io_submit and the latency of each read io_submit?

使用我们的实验代码(在2.3 GHz Haswell Macbook Pro，Ubuntu Linux 14.04上运行)，在扩展输出文件时，我们正在测量大约iouse提交的50个usecs.这太长了，我们甚至无法达到性能要求.任何帮助我以最小延迟启动写请求的指导将不胜感激.

Using our experimental code (running on a 2.3 GHz Haswell Macbook Pro, Ubuntu Linux 14.04), we are measuring about 50 usecs for a write io_submit when extending the output file. This is too long and we aren't even close to our performance requirements. Any guidance to help me launch a write request with the least latency will be greatly appreciated.

推荐答案

Linux AIO(有时称为KAIO或libaio)是一种妖术，有经验的从业者知道陷阱，但出于某种原因，与之交谈是忌讳的他们不了解的陷阱的人.通过在网络上的摸索和经验，我提出了一些示例，其中Linux通过io_submit() 提交的异步I/O可能(静默地)同步，从而将其变成阻塞调用:

Linux AIO (sometimes known as KAIO or libaio) is something of a black art where experienced practitioners know the gotchas but for some reason it's taboo to talk to someone about the gotchas they don't already know. From scratching around on the web and experience I've come up with a few examples where Linux's asynchronous I/O submission via io_submit() may become (silently) synchronous, thereby turning it into a blocking call:

  1. 您正在提交缓冲的(也称为非直接的)I/O.您受Linux缓存的约束，在以下情况下，您的提交可以同步:
    • 您正在阅读的内容已经不在读取缓存"中.
    • 写缓存"已满，直到完成一些现有的写回操作，才能接受新的写请求.
  1. You're submitting buffered (aka non-direct) I/O. You're at the mercy of Linux's caching and your submit can go synchronous when:
    • What you're reading isn't already in the "read cache".
    • The "write cache" is full and the new write request can't be accepted until some existing writeback has been completed.
  • If you submit I/Os that are "too large" (e.g. bigger than /sys/block/[disk]/queue/max_sectors_kb but the true limit may be something smaller like 512 KiB) they will be split up within the block layer and go on to chew up more than one request.
  • The system global maximum number of concurrent AIO requests (see the /proc/sys/fs/aio-max-nr documentation) can also have an impact but the result will be seen in io_setup() rather than io_submit().
  • 它需要使用一个正在使用的特定锁(例如i_rwsem).
  • 它需要分配一些额外的内存或在其中分页.

上面的列表不完整.

使用> = 4.14内核，可以使用 RWF_NONBLOCK标志以上嘈杂的一些阻塞场景.例如，当使用缓冲并尝试读取页面缓存中尚未存在的数据时，已接受或正在提议的补丁在更多可能会阻塞的情况下返回EAGAIN ，但在撰写本文时(2019年)

With >= 4.14 kernels the RWF_NONBLOCK flag can be used to make some of the blocking scenarios above noisy. For example, when using buffering and trying to read data not yet in the page cache, the RWF_NONBLOCK flag will cause submission to fail with EAGAIN when blocking would otherwise occur. Obviously you still a) need a 4.14 (or later) kernel that supports this flag and b) have to be aware of the cases it doesn't cover. I notice there are patches that have been accepted or are being proposed to return EAGAIN in more scenarios that would otherwise block but at the time of writing (2019) RWF_NONBLOCK is not supported for buffered filesystem writes.

如果您的内核> = 5.1，则可以尝试使用 io_uring ，它在不阻塞时效果更好提交(但这是一个完全不同的界面，是相当新的).

If your kernel is >=5.1, you could try using io_uring which does far better at not blocking on submission (but it's an entirely different interface and is rather new).

  • The AIOUserGuide has a "Performance considerations" section that warns about some io_submit() blocking/slowness situations.
  • A good list of Linux AIO pitfalls is given in the "Performance issues" section of the README for the ggaoed AoE target.
  • The "sleeps and waits during io_submit" XFS mailing list thread hints at some AIO queue constraints.
  • The "io_submit() blocks for writes for substantial amount of time" XFS mailing list thread has a warning from Dave Chiner that when an XFS filesystem becomes more than 85-90% full, the chances of unpredictable filesystem delays increases the closer you get to ENOSPC due to lack of large amounts of contiguous free space.
  • The "[PATCH 1/1 linux-next] ext4: add compatibility flag check to the patch" LKML thread has a reply from Ext4 lead dev Ted Ts'o talking about how filesystems can fallback to buffered I/O for O_DIRECT rather than failing the open() call.
    • In the "ubifs: Allow O_DIRECT" LKML thread Btrfs lead developer Chris Mason states Btrfs resorts to buffered I/O when O_DIRECT is requested on compressed files.
    • ZFS on Linux 0.8.0 changed ZoL's behaviour from erroring on O_DIRECT to "accepting" it by falling back to buffered I/O (see point 3 in the commit message). There's further discussion from the lead up to the commit in the ZFS on Linux "Direct IO" GitHub issue. In the "NVMe Read Performance Issues with ZFS (submit_bio to io_schedule)" issue someone suggests they are getting closer to submitting a change that enables a proper zerocopy O_DIRECT. If such a change were accepted, it would end up in some future version of ZoL greater than 0.8.2.
    • The Ext4 wiki has a warning that certain Linux implementations (Which?) fall back to buffered I/O when doing O_DIRECT allocating writes.

    相关:

    • Linux AIO: Poor Scaling
    • io_submit() blocks until a previous operation will be completed
    • buffered asynchronous file I/O on linux (but stick to the bits explicitly talking about Linux kernel AIO)

    希望这篇文章对某人有所帮助(如果有帮助，您可以投票支持吗?谢谢！).

    Hopefully this post helps someone (and if does help you could you upvote it? Thanks!).

    这篇关于Ubuntu Linux中的异步IO io_submit延迟的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
服务器开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆