多处理建议 [英] Multiprocessing Advice

问题描述

我一直在努力尝试多处理.问题是我遇到的所有示例似乎都不适合我的情况.我想进行涉及共享自变量列表的多进程或线程工作，现在我当然不希望将上述列表中的项进行两次处理，因此需要将工作划分为每个新线程/进程(或跨进程).

I've been trying to get my head around multiprocessing. The problem is all the examples I've come across don't seem to fit my scenario. I'd like to multiprocess or thread work that involves sharing a list from an argument, now of course I don't want an item from the said list being worked on twice so the work needs to be divided out to each new thread/process (or across processes).

任何有关我应该研究的方法的建议都将不胜感激.

Any advice on the approach I should be looking at would be appreciated.

我知道下面的代码无论如何都不正确，只是为了可视化我试图解释的内容.

I am aware my code below is not correct by any means, it is only to aid in visualising what I am trying to attempt to explain.

SUDO

def work_do(ip_list)
    for ip in list
        ping -c 4 ip

def mp_handler(ip_range):
    p = multiprocessing.Pool(4)
    p.map(work_do, args=(ip_range))

ip_list = [192.168.1.1-192.168.1.254]
mp_handler(ip_list)

已

某些工作代码

import multiprocessing
import subprocess

def job(ip_range):
    p = subprocess.check_output(["ping", "-c", "4", ip])
    print p

def mp_handler(ip_range):
    p = multiprocessing.Pool(2)
    p.map(job, ip_list)

ip_list = ("192.168.1.74", "192.168.1.254")

for ip in ip_list:
    mp_handler(ip)

如果运行上面的代码，您会注意到两个IP都运行了两次.如何管理仅处理列表中唯一数据的流程?

If you run the above code, you'll notice both IP's are run twice. How do I manage the processes to only work on unique data from the list?

推荐答案

使用multiprocessing可以轻松同时并发多个ip地址:

To ping multiple ip addresses concurrently is easy using multiprocessing:

#!/usr/bin/env python
from multiprocessing.pool import ThreadPool # use threads
from subprocess import check_output

def ping(ip, timeout=10):
    cmd = "ping -c4 -n -w {timeout} {ip}".format(**vars())
    try:
        result = check_output(cmd.split())
    except Exception as e:
        return ip, None, str(e)
    else:
        return ip, result, None

pool = ThreadPool(100) # no more than 100 pings at any single time
for ip, result, error in pool.imap_unordered(ping, ip_list):
    if error is None: # no error
       print(ip) # print ips that have returned 4 packets in timeout seconds

注意:我在这里使用ThreadPool作为限制并发ping次数的简便方法.如果要一次执行所有ping操作，则不需要线程或多处理模块，因为每个ping操作都已在其自己的进程中.请参见 Python中的多个ping脚本.

Note: I've used ThreadPool here as a convient way to limit number of concurrent pings. If you want to do all pings at once then you don't need neither threading nor multiprocessing modules because each ping is already in its own process. See Multiple ping script in Python.

这篇关于多处理建议的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！