Python加入一个进程而不阻塞父进程 [英] Python join a process without blocking parent

问题描述

我正在编写一个程序，它将监视特定目录中包含下载URL的新文件.一旦检测到新文件，它将在父级继续监视目录的同时创建一个新的过程来进行实际下载.我正在使用multiprocessing中的Process接口.我的问题是，除非我调用process.join()，否则子进程仍在运行，但是process.join()是一个阻止函数，无法实现创建子进程来处理实际下载的目的.

I'm writing a program that will watch a particular directory for new files containing download URLs. Once a new file is detected, it will create a new process to do the actual download while the parent continues to watch the directory. I'm using the Process interface from multiprocessing. The problem I have is that unless I call process.join() the child process is still running, but process.join() is a blocking function which defeats the purpose of creating the child to handle the actual download.

我的问题是，是否有一种以非阻塞方式加入子进程的方法，该方法将允许父进程继续做它的事情?

My question is, is there a way to join the child process in a non-blocking manner which will allow the parent to keep doing its thing?

部分代码:

def main(argv):
  # parse command line args
  ...
  # set up variables
  ...
  watch_dir(watch_dir, download_dir)


def watch_dir(wDir, dDir):
  # Grab the current watch directory listing
  before = dict([(f, None) for f in os.listdir (wDir)])

  # Loop FOREVER
  while 1:
    # sleep for 10 secs
    time.sleep(10)

    # Grab the current dir listing
    after = dict([(f, None) for f in os.listdir (wDir)])

    # Get the list of new files
    added = [f for f in after if not f in before]
    # Get the list of deleted files
    removed = [f for f in before if not f in after]

    if added:
      # We have new files, do your stuff
      print "Added: ", ", ".join(added)

      # Call the new process for downloading
      p = Process(target=child, args=(added, wDir, dDir))
      p.start()
      p.join()

    if removed:
      # tell the user the file was deleted
      print "Removed: ", ", ".join(removed)

    # Set before to the current
    before = after

def child(filename, wDir, dDir):
  # Open filename and extract the url
  ...
  # Download the file and to the dDir directory
  ...
  # Delete filename from the watch directory
  ...
  # exit cleanly
  os._exit(0)

在(据我所知)正确的p.join()之后，父级会等待子级完成执行.但这违反了创造孩子的全部目的.如果我不使用p.join()，那么孩子仍然保持活动状态，并且ps ax | grep python给我'python< defunct>'.

The parent waits for the child to finish execution before continuing after p.join() which is (as far as I can tell) correct. But that defeats the whole purpose of creating the child. If I leave off p.join() then the child remains active and a ps ax | grep python give me 'python <defunct>'.

我希望孩子完成自己的工作，然后在不牵扯父母的情况下离开.有办法吗?

I'd like the child to finish up what its doing and go away without holding up the parent. Is there a way to do it?

推荐答案

您可以设置一个单独的线程来进行连接.让它监听队列，将子进程句柄推入其中:

You can set up a separate thread which does the joining. Have it listen on a queue into which you push the subprocess handles:

class Joiner(Thread):
    def __init__(self, q):
        self.__q = q
    def run(self):
        while True:
            child = self.__q.get()
            if child == None:
                return
            child.join()

然后，执行joinq.put(p)并执行joinq.put(None)而不是p.join()，以指示线程停止.确保使用FIFO队列.

Then, instead of p.join(), do joinq.put(p) and do a joinq.put(None) to signal the thread to stop. Make sure you use a FIFO queue.

这篇关于Python加入一个进程而不阻塞父进程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！