在Google Colaboratory中长时间运行的会话结束后，如何用剩余的时间训练模型? [英] How to train model with remaining epochs after long running session has ended in Google Colaboratory.?

问题描述

我正在使用Google Colab训练具有60个历元的3D卷积神经网络，但是当达到57个历元时，我的会话就结束了.重新连接后，训练将从纪元1开始.

I am using Google Colab to train my 3D Convolutional neural network with 60 epochs but, when it reaches 57 epochs, my session is ended. After reconnecting it, the training starts from epoch 1.

在Google Colaboratory上的会话结束后，我应该怎么做才能在未完成的纪元上训练我的模型?

What should I do to train my model on uncompleted epochs after my session on Google Colaboratory has been ended?

推荐答案

常见问题解答>包含以下语句:

1. 什么是协作实验室?协作实验室是用于机器学习教育和研究的研究工具.
2. 协作室旨在用于交互使用.长期运行的背景尤其是在GPU上的计算可能会停止.... 我们鼓励希望通过运行连续或长期运行的计算的用户协作实验室的用户界面，以使用本地运行时.

1. What is Colaboratory? Colaboratory is a research tool for machine learning education and research.
2. Colaboratory is intended for interactive use. Long-running background computations, particularly on GPUs, may be stopped. ... We encourage users who wish to run continuous or long-running computations through Colaboratory’s UI to use a local runtime.

训练ML模型通常需要长时间运行的计算.因此，我正在考虑的选择是:

Training a ML model typically requires long running computations. So the options I am considering are:

1. 按照建议使用本地运行时.这可能是Cloud VM或您的笔记本电脑.
2. 使用 Cloud DataLab -您控制(并支付)Google云中的VM资源.
3. 在每个时期检查点，并将检查点和权重保存到持久性存储中.请参见 I/O食谱.如果笔记本电脑已重置，请从检查点重新开始训练.

1. Use a local runtime as suggested. This could be a Cloud VM or your laptop.
2. Use Cloud DataLab - You control (and pay for) the VM resources in the google cloud.
3. Checkpoint each epoch and save the checkpoint and weights to persistent storage. See the I/O cookbook. Then restart the training from the checkpoint if the Notebook is reset.

鉴于您的合作培训几乎已经完成，因此选择3可能是最容易上手的.这取决于您使用的是什么库，以及它们是否支持此库.

Option 3 might be the easiest to get going, given your training almost completes on Colaboratory. This depends on what libraries you are using, and whether they supports this.

如果您要运行更大的计算，请考虑使用本地运行时或DataLab.

If you are running a bigger computation then look at using local runtime or DataLab.

这篇关于在Google Colaboratory中长时间运行的会话结束后，如何用剩余的时间训练模型?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！