保存并加载模型优化器状态 [英] Save and load model optimizer state

问题描述

我正在训练一组非常复杂的模型，我正在寻找一种保存和加载模型优化器状态的方法. 教练模型"由几个其他权重模型"的不同组合组成，其中一些具有相同的权重，一些具有取决于教练的权重，等等.这个例子太复杂了，无法共享，但总之，在停止和开始训练时我无法使用model.save('model_file.h5')和keras.models.load_model('model_file.h5').

I have a set of fairly complicated models that I am training and I am looking for a way to save and load the model optimizer states. The "trainer models" consist of different combinations of several other "weight models", of which some have shared weights, some have frozen weights depending on the trainer, etc. It is a bit too complicated of an example to share, but in short, I am not able to use model.save('model_file.h5') and keras.models.load_model('model_file.h5') when stopping and starting my training.

如果训练完成，使用model.load_weights('weight_file.h5')可以很好地测试我的模型，但是如果我尝试使用此方法继续训练模型，则损失甚至无法恢复到其最后位置.我已经读到这是因为使用这种方法没有保存优化器状态，​​这很有意义.但是，我需要一种用于保存和加载训练器模型的优化器状态的方法.好像keras曾经有一个model.optimizer.get_sate()和model.optimizer.set_sate()可以满足我的要求，但现在似乎不再如此了(至少对于Adam优化器而言).当前的Keras是否还有其他解决方案?

Using model.load_weights('weight_file.h5') works fine for testing my model if the training has finished, but if I attempt to continue training the model using this method, the loss does not come even close to returning to its last location. I have read that this is because the optimizer state is not saved using this method which makes sense. However, I need a method for saving and loading the states of the optimizers of my trainer models. It seems as though keras once had a model.optimizer.get_sate() and model.optimizer.set_sate() that would accomplish what I am after, but that does not seem to be the case anymore (at least for the Adam optimizer). Are there any other solutions with the current Keras?

推荐答案

您可以从load_model和save_model函数中提取重要的行.

You can extract the important lines from the load_model and save_model functions.

要保存优化器状态，​​请在save_model中:

For saving optimizer states, in save_model:

# Save optimizer weights.
symbolic_weights = getattr(model.optimizer, 'weights')
if symbolic_weights:
    optimizer_weights_group = f.create_group('optimizer_weights')
    weight_values = K.batch_get_value(symbolic_weights)

要加载优化器状态，​​请在load_model中:

For loading optimizer states, in load_model:

# Set optimizer weights.
if 'optimizer_weights' in f:
    # Build train function (to get weight updates).
    if isinstance(model, Sequential):
        model.model._make_train_function()
    else:
        model._make_train_function()

    # ...

    try:
        model.optimizer.set_weights(optimizer_weight_values)

结合以上各行，这是一个示例:

Combining the lines above, here's an example:

1. 首先将模型拟合5个历元.

X, y = np.random.rand(100, 50), np.random.randint(2, size=100)
x = Input((50,))
out = Dense(1, activation='sigmoid')(x)
model = Model(x, out)
model.compile(optimizer='adam', loss='binary_crossentropy')
model.fit(X, y, epochs=5)

Epoch 1/5
100/100 [==============================] - 0s 4ms/step - loss: 0.7716
Epoch 2/5
100/100 [==============================] - 0s 64us/step - loss: 0.7678
Epoch 3/5
100/100 [==============================] - 0s 82us/step - loss: 0.7665
Epoch 4/5
100/100 [==============================] - 0s 56us/step - loss: 0.7647
Epoch 5/5
100/100 [==============================] - 0s 76us/step - loss: 0.7638

1. 现在保存权重和优化器状态.

model.save_weights('weights.h5')
symbolic_weights = getattr(model.optimizer, 'weights')
weight_values = K.batch_get_value(symbolic_weights)
with open('optimizer.pkl', 'wb') as f:
    pickle.dump(weight_values, f)

1. 在另一个python会话中重建模型，并加载权重.

x = Input((50,))
out = Dense(1, activation='sigmoid')(x)
model = Model(x, out)
model.compile(optimizer='adam', loss='binary_crossentropy')

model.load_weights('weights.h5')
model._make_train_function()
with open('optimizer.pkl', 'rb') as f:
    weight_values = pickle.load(f)
model.optimizer.set_weights(weight_values)

1. 继续进行模型训练.

model.fit(X, y, epochs=5)

Epoch 1/5
100/100 [==============================] - 0s 674us/step - loss: 0.7629
Epoch 2/5
100/100 [==============================] - 0s 49us/step - loss: 0.7617
Epoch 3/5
100/100 [==============================] - 0s 49us/step - loss: 0.7611
Epoch 4/5
100/100 [==============================] - 0s 55us/step - loss: 0.7601
Epoch 5/5
100/100 [==============================] - 0s 49us/step - loss: 0.7594

这篇关于保存并加载模型优化器状态的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！