在Keras中使用Adam优化器恢复培训 [英] Resume training with Adam optimizer in Keras

问题描述

我的问题很简单，但是(到目前为止)我在网上找不到明确的答案.

My question is quite straightforward but I can't find a definite answer online (so far).

在定义次数的训练后，我使用adam优化器训练的keras模型的权重已保存下来:

I have saved the weights of a keras model trained with an adam optimizer after a defined number of epochs of training using:

callback = tf.keras.callbacks.ModelCheckpoint(filepath=path, save_weights_only=True)
model.fit(X,y,callbacks=[callback])

当我关闭jupyter后恢复训练时，我可以简单地使用:

When I resume training after closing my jupyter, can I simply use:

model.load_weights(path)

继续训练.

由于Adam取决于时代数(例如学习率下降的情况)，所以我想知道在与以前相同的条件下恢复训练的最简单方法.

Since Adam is dependent on the epoch number (such as in the case of learning rate decay), I would like to know the easiest way to resume training in the same conditions as before.

按照ibarrond的回答，我写了一个小的自定义回调.

Following ibarrond's answer, I have written a small custom callback.

optim = tf.keras.optimizers.Adam()
model.compile(optimizer=optim, loss='categorical_crossentropy',metrics=['accuracy'])

weight_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path, save_weights_only=True, verbose=1, save_best_only=False)

class optim_callback(tf.keras.callbacks.Callback):
    '''Custom callback to save optimiser state'''

          def on_epoch_end(self,epoch,logs=None):
                optim_state = tf.keras.optimizers.Adam.get_config(optim)
                with open(optim_state_pkl,'wb') as f_out:                  
                       pickle.dump(optim_state,f_out)

model.fit(X,y,callbacks=[weight_callback,optim_callback()])

当我恢复训练时:

model.load_weights(checkpoint_path)
with open(optim_state_pkl,'rb') as f_out:                  
                    optim_state = pickle.load(f_out)
tf.keras.optimizers.Adam.from_config(optim_state)

我只想检查一下是否正确.再次非常感谢！

I would just like to check if this is correct. Many thanks again!!

附录:进一步阅读默认的 Keras实现亚当和原始亚当论文，我相信默认的亚当不依赖于历元，而是仅在迭代编号上.因此，这是不必要的.但是，该代码对于希望跟踪其他优化程序的任何人仍然可能有用.

Addendum: On further reading of the default Keras implementation of Adam and the original Adam paper, I believe that the default Adam is not dependent on epoch number but only on the iteration number. Therefore, this is unnecessary. However, the code may still be useful for anyone who wishes to keep track of other optimisers.

推荐答案

为了完美地捕获优化器的状态，您应该使用功能 pickle .

In order to perfectly capture the status of your optimizer, you should store its configuration using the function get_config(). This function returns a dictionary (containing the options) that can be serialized and stored in a file using pickle.

要重新启动该过程，只需d = pickle.load('my_saved_tfconf.txt')检索具有配置的字典，然后使用

To restart the process, just d = pickle.load('my_saved_tfconf.txt') to retrieve the dictionary with the configuration and then generate your Adam Optimizer using the function from_config(d) of the Keras Adam Optimizer.

这篇关于在Keras中使用Adam优化器恢复培训的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！