Keras"pickle_safe":“腌制安全"或“不可腌制"是什么意思?在Python中? [英] Keras "pickle_safe": What does it mean to be "pickle safe", or alternatively, "non picklable" in Python?

问题描述

Keras fit_generator()具有参数pickle_safe，其默认值为False.
如果是 pickle_safe，则可以更快地进行训练，并因此将标志设置为True吗?

Keras fit_generator() has a parameter pickle_safe which defaults to False.
Training can run faster if it is pickle_safe, and accordingly set the flag to True ?

根据 Kera的文档:

pickle_safe :如果为True，请使用基于进程的线程.请注意，由于此实现依赖于多处理，因此不应将不可挑选的参数传递给生成器，因为它们不能轻易地传递给子进程.

pickle_safe: If True, use process based threading. Note that because this implementation relies on multiprocessing, you should not pass non picklable arguments to the generator as they can't be passed easily to children processes.

我不明白这是什么意思.
如何确定我的论点是否为pickle_safe ?

I don't understand exactly what this is saying.
How can I determine if my arguments are pickle_safe or not ??

如果相关:
-我要传递自定义生成器
-生成器函数采用以下参数:X_train，y_train，batch_size，p_keep;
它们的类型为np.array，int，float)
-我没有使用GPU
-另外，我使用的是Keras 1.2.1，尽管我认为此参数的行为与keras 2中的行为相同

If it's relevant:
- I'm passing in a custom generator
- the generator function takes arguments: X_train, y_train, batch_size, p_keep;
they are of type np.array, int, float)
- I'm not using a GPU
- Also, I'm using Keras 1.2.1, though I believe this argument behaves the same as in keras 2

推荐答案

我对keras不熟悉，但是从文档看，pickle_safe仅表示生成器生成的元组必须可拾取". .

I have no familiarity with keras, but from a glance at the documentation, pickle_safe just means that the tuples produced by your generator must be "picklable".

pickle是一个标准的python模块，用于序列化和反序列化对象.标准的multiprocessing实现使用pickle机制在不同进程之间共享对象-由于两个进程不共享相同的地址空间，因此它们无法直接看到相同的python对象.因此，要将对象从进程A发送到进程B，将它们在A中腌制(产生以特定的众所周知格式的字节序列)，然后通过进程间通信机制将腌制的格式发送到B，然后在B中解开，在B的地址空间中产生A的原始对象的副本.

pickle is a standard python module that is used to serialize and unserialize objects. The standard multiprocessing implementation uses the pickle mechanism to share objects between different processes -- since the two processes don't share the same address space, they cannot directly see the same python objects. So, to send objects from process A to process B, they're pickled in A (which produces a sequence of bytes in a specific well-known format), the pickled format is then sent via an interprocess-communication mechanism to B, and unpickled in B, producing a copy of A's original object in B's address space.

因此，要发现您的对象是否可腌制，只需对它们调用pickle.dumps即可.

So, to discover if your objects are picklable, just invoke, say, pickle.dumps on them.

>>> import pickle
>>> class MyObject:
...    def __init__(self, a, b, c):
...      self.a = a
...      self.b = b
...      self.c = c
...
>>> foo = MyObject(1, 2, 3)
>>> pickle.dumps(foo)
b'\x80\x03c__main__\nMyObject\nq\x00)\x81q\x01}q\x02(X\x01\x00\x00\x00cq\x03K\x03X\x01\x00\x00\x00aq\x04K\x01X\x01\x00\x00\x00bq\x05K\x02ub.'
>>>

dumps产生一个字节字符串.现在，我们可以使用loads从字节字符串中将foo对象重构为bar:

dumps produces a byte string. We can now reconstitute the foo object from the byte string as bar using loads:

>>> foo_pick = pickle.dumps(foo)
>>> bar = pickle.loads(foo_pick)
>>> bar
<__main__.MyObject object at 0x7f5e262ece48>
>>> bar.a, bar.b, bar.c
(1, 2, 3)

如果某些东西无法腌制，您将得到一个例外.例如，不能腌制lambda:

If something is not picklable, you'll get an exception. For example, lambdas can't be pickled:

>>> class MyOther:
...   def __init__(self, a, b, c):
...     self.a = a
...     self.b = b
...     self.c = c
...     self.printer = lambda: print(self.a, self.b, self.c)
...
>>> other = MyOther(1, 2, 3)
>>> other_pick = pickle.dumps(other)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: Can't pickle local object 'MyOther.__init__.<locals>.<lambda>'

有关更多信息，请参阅文档: https://docs .python.org/3.5/library/pickle.html?highlight = pickle#什么可以被腌制和未腌制

See the documentation for more info: https://docs.python.org/3.5/library/pickle.html?highlight=pickle#what-can-be-pickled-and-unpickled

这篇关于Keras"pickle_safe":“腌制安全"或“不可腌制"是什么意思?在Python中?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！