在多线程管道中维护订单 [英] Maintaining Order in a Multi-Threaded Pipeline

问题描述

我正在考虑用于处理管道的多线程体系结构.我的主处理模块有一个输入队列，它从中接收数据包.然后，它对这些数据包执行转换(解密等)，并将其放入输出队列中.

I'm considering a multi-threaded architecture for a processing pipeline. My main processing module has an input queue, from which it receives data packets. It then performs transformations on these packets (decryption, etc.) and places them into an output queue.

线程进入其中，许多输入数据包可以相互独立地转换其内容.

The threading comes in where many input packets can have their contents transformed independently from one another.

但是，最重要的是输出队列必须与输入队列具有相同的顺序(即，第一个从输入队列中拉出的队列必须首先被推送到输出队列中，而不管其转换是否先完成.)

However, the punchline is that the output queue must have the same ordering as the input queue (i.e., the first pulled off the input queue must be the first pushed onto the output queue, regardless of whether its transformations finished first.)

自然地，输出队列中会有某种同步，所以我的问题是:确保这种顺序的最佳方法是什么?

Naturally, there will be some kind of synchronisation at the output queue, so my question is: what would be the best way of ensuring that this ordering is maintained?

推荐答案

具有以下假设

• 应该有一个输入队列，一个输出队列和一个工作队列
• 应该只有一个输入队列 听众
• 输出消息应包含等待 句柄和指向工作者/输出数据的指针
• 可能有任意数量的 工作线程

• there should be one input queue, one output queue and one working queue
• there should be only one input queue listener
• output message should contain a wait handle and a pointer to worker/output data
• there may be an arbitrary number of worker threads

我会考虑以下流程:

输入队列侦听器执行以下步骤:

Input queue listener does these steps:

1. 提取输入消息；
2. 创建输出消息:

1. extracts input message;
2. creates output message:

1. 初始化工作者数据结构
2. 重置等待句柄

将指向输出消息的指针排入 working 队列

将指向输出消息的指针排入 output 队列

enqueues the pointer to the output message into the working queue

enqueues the pointer to the output message into the output queue

工作线程执行以下操作:

Worker thread does the following:

1. 等待工作队列 提取指向输出的指针 来自它的消息
2. 根据给定的数据处理消息，并在完成后设置事件

1. waits on a working queue to extract a pointer to an output message from it
2. processes the message based on the given data and sets the event when done

消费者执行以下操作:

1. 等待n个输出队列 提取指向输出的指针 来自它的消息
2. 等待句柄，直到输出数据准备就绪
3. 对数据有所帮助

1. waits on n output queue to extract a pointer to an output message from it
2. waits on a handle until the output data is ready
3. does something with the data

这篇关于在多线程管道中维护订单的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！