从已经运行的Docker容器中保存工件 [英] Save artifacts from already running docker container

问题描述

我是Docker的新手。我正在使用它来训练神经网络。

I'm completely new to Docker. I'm using it to train neural networks.

我有一个正在运行的容器，执行用于训练NN的脚本，并将其权重保存在容器的可写层中。最近，我意识到这种设置是不正确的（我没有正确设置RTFM），并且在训练结束后NN权重将丢失。

I've got a running container, executing a script for training a NN, and saving its weights in container's writable layer. Recently I've realized that this setup is incorrect (I haven't properly RTFM), and the NN weights will be lost after the training finishes.

我已阅读有关卷和持久数据存储的答案和食谱。所有这些都表达了一个想法：您必须预先准备数据存储。

I've read answers and recipes about volumes and persistent data storage. All of them express one idea: you must prepare that data storage in advance.

我的容器已在运行。我了解错误的设置是我的错。无论如何，我不想丢失在执行过程中（现在正在进行中）将获得的结果。可能吗？

My container is already running. I understand that incorrect setup is my fault. Anyway, I do not want to lose results that will be obtained during this execution (that is now in progress). Is it possible?

我想到的一个解决方案是再打开一个终端并运行 watch -n 1000 docker commit< image id> tag：label

One solution that have come to my mind is to open one more terminal and run watch -n 1000 docker commit <image id> tag:label

也就是说，每1000秒提交一次快照。但是，由于历时的持续时间不同且不是1000的倍数，因此在最后一个历时获得的权重仍然处于危险之中。

That is, commit a snapshot every 1000 seconds. However, weights, obtained on the last epoch are still in danger, since epoch durations differ and are not multiple of 1000.

还有其他更优雅的解决方案吗？

Are there any more elegant solutions?

其他信息

此容器的图像是使用以下Dockerfile创建的：

Image for this container was created using the following Dockerfile:

FROM tensorflow-py3-gpu-keras

WORKDIR /root

COPY model4.py /root
COPY data_generator.py /root
COPY hyper_parameters.py /root

CMD python model4.py 

我已经从最新的tensorflow映像中手动创建了映像 tensorflow-py3-gpu-keras ，该映像是从DockerHub中提取的：

I have manually created image tensorflow-py3-gpu-keras from the latest tensorflow image, pulled from the DockerHub:

docker run tensorflow

在容器内部：

pip3安装keras

和 docker commit 在另一个终端中。

推荐答案

您是否尝试过使用 docker cp ？这样，即使容器已停止（只要尚未将其删除），也可以将文件从Docker文件系统移至主机。语法如下所示：

Have you tried using docker cp? That allows you to move files from the Docker filesystem to your host, even if the container is stopped (as long as it hasn't been removed). The syntax would look like the following:

docker cp <container id>:/path/to/file/in/container /path/to/file/in/host

这篇关于从已经运行的Docker容器中保存工件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！