处理两个传入的数据流并在 python 中组合它们? [英] Handling two incoming data streams and combining them in python?

问题描述

我一直在研究 python 中的线程、多处理异步等的各种选项，作为处理两个传入流并将它们组合的方法.关于的信息很多，但示例往往令人费解和复杂，更常见的是将单个任务拆分为多个线程或进程以加快任务的最终结果.

I have been researching various options in python of threading, multiprocessing async etc as ways of handling two incoming streams and combining them. There is a lot of information about, but the examples are often convoluted and complicated, and more commonly are to split up a single task into multiple threads or processes to speed up the end result of the task.

我有一个通过套接字传入的数据流(目前使用 UDP 作为在我的 PC 上本地运行的另一个应用程序，但如果应用程序需要在单独的 PC 上运行，将来可能会考虑切换到 TCP)，并且通过 RS232 适配器传入的串行流，我需要合并这些流.然后在另一个套接字上重新传输这个新流.

I have a data stream coming in over a socket (currently using UDP as its another application running locally on my PC, but may consider switching to TCP in the future if the application needs to be run on a separate PC), and a serial stream coming in via an RS232 adaptor, and I need to combine the streams. This new stream is then retransmitted on another socket.

问题是它们以不同的速率进入(串行数据以 125hz 进入，套接字数据以 60-120hz 进入)，所以我想将最新的串行数据添加到套接字数据中.

The issue is that they come in at different rates (serial data is coming in at 125hz, socket data at 60-120hz), so I want to add the latest serial data to the socket data.

我的问题本质上是根据其他人以前的经验处理这个问题的最佳方法是什么.由于这本质上是一个 I/O 任务，因此它更倾向于线程(我知道 GIL 仅限于并发)，但由于输入速率高，我想知道多处理是否可行?

My question is essentially what is the best way to handle this, based on other peoples previous experience. Since this is essentially an I/O task, it lends more towards threading (which I know is limited to concurrency by the GIL), but due to the high input rate, I am wondering if Multi-processing is the way to go?

如果使用线程，我想访问每个共享资源的最佳方法是使用锁将串行数据写入对象，并在有新套接字数据时在单独的线程中获取锁，访问最新的串行对象中的数据，处理它然后在另一个套接字上发送它.但是，主线程在每个新传入的套接字消息之间有很多工作要做.

If using threading, I guess the best way to access each shared resource is using a Lock to write the serial data to an object, and in a separate thread whenever there is new socket data then acquiring the lock, accessing the latest serial data in the object, processing it then sending it out on the other socket. However, the main thread has a lot of work to in between each new incoming socket message.

通过多处理，我可以使用管道从另一个进程请求和接收最新的串行数据，但这只会卸载串行数据处理，并且仍然为主进程留下很多.

With Multi-processing I could use a pipe to request and receive the latest serial data from the other process, but that only offloads the serial data handling, and still leaves a lot for the main process.

推荐答案

我认为使用 选择 非常简单.它告诉你哪个套接字有数据(或 EOF)要读取.

I think using select is very straightforward. It tells you which socket has data (or EOF) to read.

其实之前也有人问过类似的问题:Python - 服务器从两个 UDP 套接字监听

Actually, a similar question has been asked before: Python - Server listening from two UDP sockets

请注意，只有从 select 返回的套接字中读取的内容才能保证不会阻塞.在继续阅读之前再次检查.这意味着如果您正在读取数据流，请读入缓冲区，直到您收到整行或其他可以处理的数据单元.

Please note that only one read from a socket returned by select is guaranteed not to block. Check again before continuing reading. That means if you are reading a data stream, read into a buffer until you receive a whole line or other data unit that can be processed.

您的问题与链接的问题不同，因为您需要从网络和串行接口读取数据.Linux 没有问题，任何文件描述符都可以与 select 一起使用.但是在 Windows 上，只有套接字可以与 select 一起使用.我不使用 Windows，但看起来您需要一个专用线程来读取串行线路.

Your question differs from the linked one, because you need to read from network and a serial interface. Linux has no problem with it, any file descriptor can be used with select. However on Windows, only sockets can be used with select. I do not work with Windows, but it looks like you will need a dedicated thread for reading the serial line.

这篇关于处理两个传入的数据流并在 python 中组合它们?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！