关于keras.utils.Sequence的澄清 [英] Clarification about keras.utils.Sequence

问题描述

Keras几乎没有关于 keras.utils.Sequence 的信息，这实际上是我想要获得我的唯一原因。 keras.utils.Sequence 的批处理生成器是我不想自己写带有队列的线程池，但是我不确定这是否是我的任务的最佳选择，这是我的问题：

Keras have very little info about keras.utils.Sequence, actually the only reason I want to derive my batch generator from keras.utils.Sequence is that I want to not to write thread pool with queue by myself, but I'm not sure if it's best choice for my task, here is my questions:

1. 如果我有随机生成器，并且 __ len __ 应该返回什么？我没有
   有样本的预定义列表。


2. keras.utils.Sequence
   应该与 fit_generator ，我对
max_queue_size ， workers ， use_multiprocessing ，随机播放
参数。


3. 在喀拉拉邦还有哪些其他选择？

1. What should __len__ return if I have random generator and I don't have any predefined 'list' with samples.
2. How keras.utils.Sequence should be used with fit_generator, I'm interested in max_queue_size,workers, use_multiprocessing, shuffle parameters mostly.
3. What are other options avalible in keras?

推荐答案



1. 您想要的任何东西，考虑到一个纪元将得到<$ c序列中的$ c> len 个批次。


2. 没有秘密，可以将其用作任何其他生成器，不同之处在于您可以执行 steps_per_epoch = len（generator）或 steps_per_epoch =无。
   
   
   
   
   • max_queue_size ：任何值，这将加载将在内存中等待直到它们转到的批处理进入模型
   
   
   • workers ：任何值，这将是并行线程的数量（如果名称为不精确）将要批量加载
   
   
   • use_multiprocessing ：我不知道这一点。对我来说不是必须的，而且我唯一的尝试是遇到很多问题，足以冻结我的机器
   
   
   • shuffle ：来自文档：布尔值。是否在每个纪元开始时对批次顺序进行洗牌。仅用于Sequence实例（keras.utils.Sequence）。当steps_per_epoch不为None时无效。
   
   

1. Anything you want, considering that one epoch will get len batches from the Sequence.
2. There is no secret, use it as any other generator, with the difference that you may do steps_per_epoch=len(generator) or steps_per_epoch=None.
   • max_queue_size: any value, this will load batches that will be waiting in memory until their turn to get into the model
   • workers: any value, this will be the number of parallel "threads" (forgive me if the name is not precise) that will be loading batches
   • use_multiprocessing: I don't know this one. It was not necessary for me and the only time I tried it was buggy enough to freeze my machine
   • shuffle: From the documentation: Boolean. Whether to shuffle the order of the batches at the beginning of each epoch. Only used with instances of Sequence (keras.utils.Sequence). Has no effect when steps_per_epoch is not None.

在常规生成器上 Sequence 的优点：

Advantages of Sequence over a regular generator:

通过序列，可以跟踪已提取的批次，将哪些批次发送到哪个线程进行加载，并且永远不会发生冲突，因为基于索引。

With sequence, it's possible to keep track of which batches were already taken, which batches are sent to which thread for loading, and there will never be a conflict because it's based on indices.

使用生成器，并行处理将无法跟踪已经采摘了哪些批次，因为线程之间不互相交谈，除了通过批次产生批次，别无选择

With generator, parallel processing will lose track of what batches were already taken or not because threads don't talk to each other and there is no other option than yielding batch by batch sequentially.

生成器和序列在循环中的优点

循环中，您将等待批处理加载，等待模型培训，等待批处理加载，等待模型培训。

In a loop, you will "wait for batch load", "wait for model training", "wait for batch load", "wait for model training".

使用 fit_generator ，将在模型训练时批量加载批处理，这两种情况同时发生。

With fit_generator, batches will be loaded "while" the model is training, you have both things happening simultaneously.

对于非常简单的生成器来说，不会有太大的影响。对于复杂的生成器，增强器，大图像加载器等，生成时间非常重要，并且可能严重影响您的速度。

For very simple generators, there won't be a big impact. For complex generators, augmentators, big image loaders, etc., the generation time is very significant and may severely impact your speed.

这篇关于关于keras.utils.Sequence的澄清的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！