如何将python asyncio与线程结合在一起? [英] How to combine python asyncio with threads?

问题描述

我已经使用以下命令成功构建了 RESTful微服务 Python asyncio和aiohttp，它们侦听POST事件以收集来自各种供稿器的实时事件.

I have successfully built a RESTful microservice with Python asyncio and aiohttp that listens to a POST event to collect realtime events from various feeders.

然后，它构建了一个内存结构，以将事件的最后24小时缓存在嵌套的defaultdict/deque结构中.

It then builds an in-memory structure to cache the last 24h of events in a nested defaultdict/deque structure.

现在，我想定期检查该结构到磁盘的位置，最好使用泡菜.

Now I would like to periodically checkpoint that structure to disc, preferably using pickle.

由于内存结构可以大于100MB，所以我想避免在检查点结构所需的时间上暂停传入的事件处理.

Since the memory structure can be >100MB I would like to avoid holding up my incoming event processing for the time it takes to checkpoint the structure.

我宁愿为该结构创建快照副本(例如，Deepcopy)，然后花点时间将其写入磁盘并按预设的时间间隔重复进行.

I'd rather create a snapshot copy (e.g. deepcopy) of the structure and then take my time to write it to disk and repeat on a preset time interval.

为此，我一直在寻找有关如何组合线程(甚至是最佳解决方案的线程?)和asyncio的示例，但是找不到对我有用的东西.

I have been searching for examples on how to combine threads (and is a thread even the best solution for this?) and asyncio for that purpose but could not find something that would help me.

非常感谢任何入门指南！

Any pointers to get started are much appreciated!

推荐答案

使用对于使用ProcessPoolExecutor还是ThreadPoolExecutor，这很难说.腌制一个大对象肯定会占用一些CPU周期，起初您会认为ProcessPoolExecutor是可行的方法.但是，要将100MB对象传递到池中的Process，则需要在主进程中对实例进行酸洗，然后通过IPC将字节发送给子进程，在子进程中将其解酸，然后再次对其进行酸洗 ，因此您可以将其写入磁盘.鉴于此，我的猜测是，即使由于GIL而导致性能下降，酸洗/去酸洗的开销也将足够大，以使您最好使用ThreadPoolExecutor.

As for whether to use a ProcessPoolExecutor or ThreadPoolExecutor, that's kind of hard to say; pickling a large object will definitely eat some CPU cycles, which initially would you make think ProcessPoolExecutor is the way to go. However, passing your 100MB object to a Process in the pool would require pickling the instance in your main process, sending the bytes to the child process via IPC, unpickling it in the child, and then pickling it again so you can write it to disk. Given that, my guess is the pickling/unpickling overhead will be large enough that you're better off using a ThreadPoolExecutor, even though you're going to take a performance hit because of the GIL.

也就是说，对两种方法进行测试并确定确定非常简单，因此您也可以这样做.

That said, it's very simple to test both ways and find out for sure, so you might as well do that.

这篇关于如何将python asyncio与线程结合在一起?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！