关于 Keras fit_generator() 中的多处理和工作人员与 spyder 中的 windows 10 的混淆 [英] Confusion about multiprocessing and workers in Keras fit_generator() with windows 10 in spyder

问题描述

在 fit_generator() 的文档中(文档:https://keras.io/models/sequential/#fit_generator) 它表示参数 use_multiprocessing 接受一个布尔值，如果设置为 True 则允许基于进程的线程.

In the documentation for fit_generator() (docs: https://keras.io/models/sequential/#fit_generator) it says that the parameter use_multiprocessing accepts a bool that if set to True allows process-based threading.

它还说参数workers是一个整数，它指定如果使用基于进程的线程，要启动多少进程.显然它默认为 1(基于单个进程的线程)，如果设置为 0，它将在主线程上执行生成器.

It also says that the parameter workers is an integer that designates how many process to spin up if using process-based threading. Apparently it defaults to 1 (a single process based thread) and if set to 0 it will execute the generator on the main thread.

我认为这意味着如果 use_multiprocessing=True 并且 workers > 0(我们以 6 为例)，它将启动 6 个独立运行生成器的进程.但是，当我对此进行测试时，我认为我一定是误解了某些东西(见下文).

What I thought this meant was that if use_multiprocessing=True and workers > 0 (let's use 6 for an example) that it would spin up 6 processes running the generator independently. However, when I test this I think I must be misunderstanding something (see below).

我的困惑源于这样一个事实，即如果我将 use_multiprocessing 设置为 False 并且 workers = 1，那么在我的任务管理器中，我可以看到我的所有 12 个虚拟内核都被均匀地利用了，而我的 CPU 使用率大约是 50%训练我的模型(作为参考，我有一个支持虚拟化的 6 核 i7-8750H CPU，并且我在 BIOS 中启用了虚拟化).如果我增加工作人员的数量，CPU 使用率会达到 100%，训练速度会更快.如果我将工作线程的数量减少到 0 以使其在主线程上运行，我可以看到我的所有虚拟内核仍在使用中，但似乎有些不平衡，CPU 使用率约为 36%.

My confusion arises from the fact that if I set use_multiprocessing to False and workers = 1 then in my task manager I can see that all 12 of my virtual cores are being utilized somewhat evenly and I am at about 50% CPU usage while training my model (for reference, I have an i7-8750H CPU with 6 cores that support virtualization and I have virtualization enabled in BIOS). If I increase the number of workers, the CPU usage goes to 100% and training is much faster. If I decrease the number of workers to 0 so that it runs on the main thread, I can see that all of my virtual cores are still being used, but it seems somewhat uneven and CPU usage is at about 36%.

不幸的是，如果我将 multiprocessing = True 设置为 True，则会收到损坏的管道错误.我还没有解决这个问题，但我想更好地了解我在这里要解决的问题.

Unfortunately, if I set multiprocessing = True, then I get a brokenpipe error. I have yet to fix this, but I'd like to better understand what I am trying to fix here.

如果有人可以解释使用 use_multiprocessing = True 和 use_multiprocessing = False 进行训练之间的区别，以及当工人 = 0、1 和 >1 时，我将非常感激.如果重要的话，我将使用 tensorflow(gpu 版本)作为带有 IPython 控制台的 Spyder 中带有 python 3.6 的 keras 的后端.

If someone could please explain the difference between training with use_multiprocessing = True and use_multiprocessing = False, as well as when workers are = 0, 1, and >1 I would be very grateful. If it matters, I am using tensorflow (gpu version) as the backend for keras with python 3.6 in Spyder with the IPython Console.

我的怀疑是 use_multiprocessing 实际上是在 True 时启用多处理，而 workers>1 当 use_multiprocessing=False 时正在设置线程数，但这只是一个猜测.

My suspicion is that use_multiprocessing is actually enabling multiprocessing when True whereas workers>1 when use_multiprocessing=False is setting the number of threads, but that's just a guess.

推荐答案

我唯一知道的是，当 use_multiprocessing=False 和 workers >1，有很多并行的数据加载线程(这些名字，线程，进程等我不是很擅长).但是有五个并行的前端将数据加载到队列中(因此，加载数据更快，但不会影响模型的速度 - 当数据加载时间过长时这可能会很好).

The only thing I know is that when use_multiprocessing=False and workers > 1, there are many parallel data loading threads (I'm not really good with these names, threads, processes, etc.). But there are five parallel fronts loading data to the queue (so, loading data is faster, but it doesn't affect the model's speed - this can be good when data loading takes too long).

每当我尝试 use_multiprocessing=True 时，一切都被冻结了.

Whenever I tried use_multiprocessing=True, everything got frozen.

这篇关于关于 Keras fit_generator() 中的多处理和工作人员与 spyder 中的 windows 10 的混淆的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！