posix管道作为工作队列 [英] posix pipe as a work queue

问题描述

我所见过的工作队列的常规实现涉及互斥锁和条件变量.

The normal implementations of a work queue I have seen involve mutexes and condition variables.

A) Acquires Lock
B) While Queue empty
      Wait on Condition Variable (thus suspending thread and releasing lock)
C) Work object retrieved from queue
D) Lock is released
E) Do Work
F) GOTO A

制作人:

A) Acquires Lock
B) Work is added to queue
C) condition variable is signaled (potentially releasing worker)
D) Lock is released

我一直在浏览一些代码，并且看到了使用POSIX管道的实现(以前没有见过这种技术).

I have been browsing some code and I saw an implementation using POSIX pipes (I have not seen this technique before).

A) Do select on pipe (thus suspending thread while no work)
B) Get Job from pipe
C) Do Work
D) GOTO A

制作人:

A) Write Job to pipe.

由于生产者和使用者是同一应用程序内的线程(因此它们共享相同的地址空间，因此它们之间的指针有效)；作业将作为工作对象(一个C ++对象)的地址写入管道.因此，必须从管道写入的所有内容都是一个8字节的地址.

Since the producer and consumer are threads inside the same application (thus they share the same address space and thus pointers between them are valid); the jobs are written to the pipe as the address of the work object (A C++ object). So all that has to be written/read from the pipe is an 8-byte address.

我的问题是:

• 这是一种常见的技术吗(我对此有所庇护)，优点/缺点是什么?

我的好奇心激起了，因为管道技术不涉及任何可见的锁定或信号(它可能隐藏在选择中).所以我想知道这是否会更有效?

My curiosity was piqued because the pipe technique does not involve any visible lock or signals (it may be hidden in the select). So I was wondering if this would be more efficient?

基于@Maxim Yegorushkin答案中的评论.

Based on comments in @Maxim Yegorushkin answer.

实际上，这种情况下的生产者"涉及大量并行来源的大量高容量IO.因此，我怀疑原始作者虽然非常希望该线程在任何情况下都不会阻塞，但也不想在生产者"线程中进行高成本的工作.

Actually the "Producer" in this scenario is involved in a lot of high volume IO from lots of source in parallel. So I suspect that the original author though it very desirable that this thread did not block under any circumstances, but also did not want to high cost work in the "Producer" thread.

推荐答案

正如已经在这里提到的，人们使用管道作为队列，以避免在非阻塞I/O线程(即处理select/epoll上的多个套接字和块).如果I/O线程在条件变量或互斥锁上阻塞，则无法再进行非阻塞I/O.

As it's been mentioned here already, people use pipes as queues to avoid blocking on a condition variable in a non-blocking I/O thread (i.e. the thread that handles multiple sockets and blocks on select/epoll). If an I/O thread blocks on a condition variable or a mutex it can't do non-blocking I/O any more.

有人说，写入管道涉及系统调用，并且当线程间事件的数量很高时，可能会增加延迟.这仅适用于基于朴素的基于管道的队列实现.

Some say that writing into a pipe involves a system call and may increase latency when the volume of inter-thread events is high. That is only true for naive pipe-based queue implementations.

高级实现使用作业/事件的无锁链接列表，仅当将第一个作业添加到列表时，才将管道写入以从阻塞的epoll调用中唤醒目标I/O线程(本质上是使用管道)作为边缘触发的通知机制，但不用于将指针传递给作业/事件).因为唤醒线程需要花费几微秒的时间，所以在此期间可能会有更多的作业/事件发布到该线程的事件队列中，但是每个后续事件都不需要写入管道，直到稍后I/O线程唤醒并消耗队列中的所有事件.另外，在较新的Linux内核中，可以使用更快的eventfd代替管道来唤醒I/O线程.

Advanced implementations use lock-free linked lists of jobs/events and only when the first job is added to the list the pipe is written to to wake the target I/O thread from the blocking epoll call (essentially using pipe as an edge-triggered notification mechanism but not for passing pointers to jobs/events). Because it takes a few micro-seconds to wake up a thread there may be more jobs/events posted to that thread's event queue during this time but every subsequent event doesn't require writing to the pipe, until later time when the I/O thread wakes up and consumes all events in the queue. Also, in newer Linux kernel a faster eventfd can be used instead of pipe to wake up an I/O thread.

这篇关于posix管道作为工作队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！