使用Python多处理在工作人员之间共享变量 [英] Share a variable between workers with Python multiprocessing

问题描述

如何在Python中读取和更新多个工作人员之间共享的变量?

How can I read and update a variable shared between multiple workers in Python?

例如，我正在使用Python中的多个进程扫描文件列表，并希望检查父目录是否已被扫描.

For example, I'm scanning through a list of files using multiple processes in Python, and would like to check if the parent directory has been scanned or not.

def readFile(filename):
  """ Add the parent folder to the database and process the file
  """

  path_parts = os.path.split(filename)
  dirname = os.path.basename(path_parts[0])
  if dirname not in shared_variable:
    # Insert into the database


   #Other file functions


def main():
  """ Walk through files and pass each file to readFile()
  """
  queue = multiprocessing.Queue()
  pool = multiprocessing.Pool(None, init, [queue])

  for dirpath, dirnames, filenames in os.walk(PATH):

    full_path_fnames = map(lambda fn: os.path.join(dirpath, fn),
                           filenames)
    pool.map(readFile, full_path_fnames)

推荐答案

您可以使用multiprocessing.Manager来帮助解决此问题.它允许您创建一个可以在进程之间共享的列表:

You can use multiprocessing.Manager to help with this. It allows you to create a list that can be shared between processes:

from functools import partial
import multiprocessing

def readFile(shared_variable, filename):
  """ Add the parent folder to the database and process the file
  """

  path_parts = os.path.split(filename)
  dirname = os.path.basename(path_parts[0])
  if dirname not in shared_variable:
    # Insert into the database


   #Other file functions


def main():
  """ Walk through files and pass each file to readFile()
  """
  manager = multiprocessing.Manager()
  shared_variable = manager.list()
  queue = multiprocessing.Queue()
  pool = multiprocessing.Pool(None, init, [queue])

  func = partial(readFile, shared_variable)
  for dirpath, dirnames, filenames in os.walk(PATH):

    full_path_fnames = map(lambda fn: os.path.join(dirpath, fn),
                           filenames)
    pool.map(func, full_path_fnames)

partial仅用于使shared_variable以及full_path_fnames的每个成员通过map传递给readFile的每个调用变得更容易.

The partial is just used to make it easier to pass shared_variable to each call of readFile, along with each member of full_path_fnames via map.

这篇关于使用Python多处理在工作人员之间共享变量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！