我可以同时运行两个多线程函数吗? [英] Can I have two multithreaded functions running at the same time?

问题描述

我对多线程非常陌生.我的python脚本中有2个函数.一个函数enqueue_tasks遍历大量小项目，并对每个项目执行一项任务，该任务涉及将一个项目追加到列表中(我们称其为master_list).我已经使用期货使用了多线程.

I'm very new to multi-threading. I have 2 functions in my python script. One function enqueue_tasks iterates through a large list of small items and performs a task on each item which involves appending an item to a list (lets call it master_list). This I already have multi-threaded using futures.

executor = concurrent.futures.ThreadPoolExecutor(15) # Arbitrarily 15
futures = [executor.submit(enqueue_tasks, group) for group in grouper(key_list, 50)]
concurrent.futures.wait(futures)

我还有另一个功能process_master，它遍历上面的master_list并检查列表中每个项目的状态，然后执行一些操作.

I have another function process_master that iterates through the master_list above and checks the status of each item in the list, then does some operation.

我可以使用与上述相同的方法对process_master使用多线程吗?此外，我可以让它与enqueue_tasks同时运行吗?这意味着什么? process_master依赖于enqueue_tasks中的列表，因此同时运行它们会成为问题吗?有什么办法可以延迟第二个功能吗? (也许使用time.sleep)?

Can I use the same method above to use multi-threading for process_master? Furthermore, can I have it running at the same time as enqueue_tasks? What are the implications of this? process_master is dependent on the list from enqueue_tasks, so will running them at the same time be a problem? Is there a way I can delay the second function a bit? (using time.sleep perhaps)?

推荐答案

不，这是不安全的.如果enqueue_tasks和process_master同时运行，则可能同时在process_master对其进行迭代的同时将项目添加到enqueue_tasks内部的master_list中.在迭代时更改Iterable的大小会在Python中引起未定义的行为，应始终避免.您应该使用threading.Lock保护将项目添加到master_list的代码，以及在master_list上迭代的代码，以确保它们永远不会同时运行.

No, this isn't safe. If enqueue_tasks and process_master are running at the same time, you could potentially be adding items to master_list inside enqueue_tasks at the same time process_master is iterating over it. Changing the size of an iterable while you iterate over it causes undefined behavior in Python, and should always be avoided. You should use a threading.Lock to protect the code that adds items to master_list, as well as the code that iterates over master_list, to ensure they never run at the same time.

更好的方法是使用 Queue.Queue ()，而不是list，后者是线程安全的数据结构.将项目添加到enqueue_tasks中的Queue，并添加get中Queue中的get项目.这样，process_master可以安全地与enqueue_tasks同时运行.

Better yet, use a Queue.Queue (queue.Queue in Python 3.x) instead of a list, which is a thread-safe data structure. Add items to the Queue in enqueue_tasks, and get items from the Queue in process_master. That way process_master can safely run a the same time as enqueue_tasks.

这篇关于我可以同时运行两个多线程函数吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！