Python中多处理或多线程的动态刷新打印 [英] Dynamic refresh printing of multiprocessing or multithreading in Python

问题描述

我已经实现了一个多处理下载器.如何打印可以自动刷新的状态栏(完成率、下载速度)在终端的不同部分.

I have implemented a multiprocessing downloader. How can I print the status bar (complete rate, download speed) which can refresh automatically in different part on the terminal.

像这样:

    499712  [6.79%]   68k/s     // keep refreshing
    122712  [16.79%]   42k/s    // different process/thread
     99712  [56.32%]   10k/s

代码:

download(...)
...
    f = open(tmp_file_path, 'wb')
    print "Downloading: %s Bytes: %s" % (self.file_name, self.file_size)
    file_size_dl = 0
    block_sz = 8192
    start_time = time.time()
    while True:
        buffer = self.opening.read(block_sz)
        if not buffer:
            break

        file_size_dl += len(buffer)
        f.write(buffer)
        end_time = time.time()
        cost_time = end_time - start_time
        if cost_time == 0:
            cost_time = 1
        status = "
%10d  [%3.2f%%]  %3dk/s" % (file_size_dl,
                file_size_dl * 100. / self.file_size,
                file_size_dl * 100. / 1024 / 1024 / cost_time)
        print status,
        sys.stdout.flush()
    f.close()

DownloadProcess 继承 Process 类并触发下载方法.

DownloadProcess inherits Process class and trigger the download method.

我使用队列来存储 url.这是启动过程

I use queue to store the url. Here is starting process

  ...
  for i in range(3):
    t = DownloadProcess(queue)
    t.start()
    for url in urls:
        queue.put(url)
  queue.join()

推荐答案

下面是一个实现了多处理和多线程的演示.要尝试其中一个，只需取消注释代码顶部的导入行即可.如果您在一行上有一个进度条，那么您可以使用打印 ' ' 的技术将光标移回该行的开头.但是如果你想拥有多行进度条，那么你将不得不变得更加狂热.每次我想打印进度条时，我都只是清除了屏幕.查看文章 Python 中 Unix 上的控制台输出 它对我生成下面的代码有很大帮助.他展示了这两种技术.您还可以试一试作为 python 标准库一部分的curses 库.问题 多行进度条提出了类似的问题.主线程/进程产生执行工作的子线程，并使用队列将它们的进度返回给主线程.我强烈建议使用队列进行进程间/线程通信.主线程然后显示进度并在退出之前等待所有子线程结束执行.

Below is a demo that has implemented both multi-processing and multi-threading. To try one or the other just uncomment the import lines at the top of the code. If you have a progress bar on a single line then you can use the technique that you have of printing ' ' to move the cursor back to the start of the line. But if you want to have multi-line progress bars then you are going to have to get a little fancier. I just cleared the screen each time I wanted to print the progress bars. Check out the article console output on Unix in Python it helped me a great deal in producing the code below. He shows both techniques. You can also give the curses library that is part of python standard library a shot. The question Multiline progress bars asks a similar thing. The main thread/process spawns the child threads that do the work and communicate their progress back to the main thread using a queue. I highly recommend using queues for inter-process/thread communication. The main thread then displays the progress and waits for all children to end execution before exiting itself.

代码

import time, random, sys, collections
from multiprocessing import Process as Task, Queue
#from threading import Thread as Task
#from Queue import Queue

def download(status, filename):
    count = random.randint(5, 30)
    for i in range(count):
        status.put([filename, (i+1.0)/count])
        time.sleep(0.1)

def print_progress(progress):
    sys.stdout.write('33[2J33[H') #clear screen
    for filename, percent in progress.items():
        bar = ('=' * int(percent * 20)).ljust(20)
        percent = int(percent * 100)
        sys.stdout.write("%s [%s] %s%%
" % (filename, bar, percent))
    sys.stdout.flush()

def main():
    status = Queue()
    progress = collections.OrderedDict()
    workers = []
    for filename in ['test1.txt', 'test2.txt', 'test3.txt']:
        child = Task(target=download, args=(status, filename))
        child.start()
        workers.append(child)
        progress[filename] = 0.0
    while any(i.is_alive() for i in workers):
        time.sleep(0.1)
        while not status.empty():
            filename, percent = status.get()
            progress[filename] = percent
            print_progress(progress)
    print 'all downloads complete'

main()

演示

这篇关于Python中多处理或多线程的动态刷新打印的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！