在Docker构建中使用pip缓存目录 [英] Using a pip cache directory in docker builds
问题描述
我希望尽快将pip install指令放入我的docker build内.
我已阅读很多 帖子解释如何在其余部分之前添加requirements.txt如果您的requirements.txt保持不变，该应用程序的帮助可帮助您利用Docker自己的图像缓存.但这对依赖项的更改根本没有帮助，即使更改很小.
下一步是如果我们可以使用一致的pip缓存目录.默认情况下，pip将在~/.cache/pip中缓存下载的软件包(在Linux上)，因此，如果您要安装的模块的版本与系统上任何位置之前安装的模块相同，则无需并再次下载，而只是使用缓存的版本.如果我们可以将共享的缓存目录用于Docker构建，则可以大大加快依赖项的安装速度.
但是，似乎没有任何简单的方法可以在运行docker build时挂载卷.构建环境似乎基本上是不可渗透的.我发现一篇文章建议在主机上运行rsync服务器，然后在内部进行破解以获取主机IP的天才却又复杂的方法，即从主机中同步pip缓存.但是我不喜欢在Jenkins(最好的时候这不是最安全的平台)上运行rsync服务器的想法.
有人知道还有其他方法可以更简单地实现共享缓存量吗?
我建议您使用此.
Dockerfile:
# syntax = docker/dockerfile:experimental
FROM python:3.6-alpine
RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml
注意:# syntax = docker/dockerfile:experimental是必须的，您必须在Dockerfile的开头添加它才能启用此功能.
1.
首次执行构建:
export DOCKER_BUILDKIT=1
docker build --progress=plain -t abc:1 . --no-cache
第一个日志:
#9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install...
#9 digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5
#9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml"
#9 started: 2019-09-20 03:11:35.296107357 +0000 UTC
#9 1.955 Collecting pyyaml
#9 3.050 Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)
#9 5.006 Building wheels for collected packages: pyyaml
#9 5.007 Building wheel for pyyaml (setup.py): started
#9 5.249 Building wheel for pyyaml (setup.py): finished with status 'done'
#9 5.250 Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=867daf35eab43c2d047ad737ea1e9eaeb4168b87501cd4d62c533f671208acaa
#9 5.250 Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030
#9 5.267 Successfully built pyyaml
#9 5.274 Installing collected packages: pyyaml
#9 5.309 Successfully installed pyyaml-5.1.2
#9completed: 2019-09-20 03:11:42.221146294 +0000 UTC
#9 duration: 6.925038937s
从上面可以看到，该版本将首次从Internet下载pyyaml.
2.
第二个执行版本:
docker build --progress=plain -t abc:1 . --no-cache
第二条日志:
#9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install...
#9 digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5
#9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml"
#9 started: 2019-09-20 03:16:58.588157354 +0000 UTC
#9 1.786 Collecting pyyaml
#9 2.234 Installing collected packages: pyyaml
#9 2.270 Successfully installed pyyaml-5.1.2
#9completed: 2019-09-20 03:17:01.933398002 +0000 UTC
#9 duration: 3.345240648s
从上方，您可以看到该构建不再从Internet下载程序包，只需使用缓存即可.注意，这不是传统的docker build缓存，因为我使用的是--no-cache，而是我安装到build中的/root/.cache/pip.
3.
第三个执行构建，该构建删除了构建包缓存:
docker builder prune
docker build --progress=plain -t abc:1 . --no-cache
第三条日志:
#9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install...
#9 digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5
#9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml"
#9 started: 2019-09-20 03:19:07.434792944 +0000 UTC
#9 1.894 Collecting pyyaml
#9 2.740 Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)
#9 3.319 Building wheels for collected packages: pyyaml
#9 3.319 Building wheel for pyyaml (setup.py): started
#9 3.560 Building wheel for pyyaml (setup.py): finished with status 'done'
#9 3.560 Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=cea5bc4689e231df7915c2fc3abca225d4ee2e869a7540682aacb6d42eb17053
#9 3.560 Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030
#9 3.580 Successfully built pyyaml
#9 3.585 Installing collected packages: pyyaml
#9 3.622 Successfully installed pyyaml-5.1.2
#9completed: 2019-09-20 03:19:12.530742712 +0000 UTC
#9 duration: 5.095949768s
从上方，您可以查看是否删除buildkit缓存，然后再次下载软件包.
简而言之，它将为您提供多次构建之间的共享缓存，并且仅在映像构建时才会挂载此缓存.但是，图像本身不会具有这些缓存，因此请避免在图像中使用很多中间层.I'm hoping to get my pip install instructions inside my docker builds as fast as possible.
I've read many posts explaining how adding your requirements.txt before the rest of the app helps you take advantage of Docker's own image cache if your requirements.txt hasn't changed. But this is no help at all when dependencies do change, even slightly.
The next step would be if we could use a consistent pip cache directory. By default, pip will cache downloaded packages in ~/.cache/pip (on Linux), and so if you're ever installing the same version of a module that has been installed before anywhere on the system, it shouldn't need to go and download it again, but instead simply use the cached version. If we could leverage a shared cache directory for docker builds, this could help speed up dependency installs a lot.
However, there doesn't appear to be any simple way to mount a volume while running docker build. The build environment seems to be basically impenetrable. I found one article suggesting a genius but complex method of running an rsync server on the host and then, with a hack inside the build to get the host IP, rsyncing the pip cache in from the host. But I'm not relishing the idea of running an rsync server in Jenkins (which isn't the most secure platform at the best of times).
Does anyone know if there's any other way to achieve a shared cache volume more simply?
I suggest you to use buildkit, also see this.
Dockerfile:
# syntax = docker/dockerfile:experimental
FROM python:3.6-alpine
RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml
NOTE: # syntax = docker/dockerfile:experimental is a must，you have to add it at the beginning of Dockerfile to enable this feature.
1.
The first execute build:
export DOCKER_BUILDKIT=1
docker build --progress=plain -t abc:1 . --no-cache
The first log:
#9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install...
#9 digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5
#9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml"
#9 started: 2019-09-20 03:11:35.296107357 +0000 UTC
#9 1.955 Collecting pyyaml
#9 3.050 Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)
#9 5.006 Building wheels for collected packages: pyyaml
#9 5.007 Building wheel for pyyaml (setup.py): started
#9 5.249 Building wheel for pyyaml (setup.py): finished with status 'done'
#9 5.250 Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=867daf35eab43c2d047ad737ea1e9eaeb4168b87501cd4d62c533f671208acaa
#9 5.250 Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030
#9 5.267 Successfully built pyyaml
#9 5.274 Installing collected packages: pyyaml
#9 5.309 Successfully installed pyyaml-5.1.2
#9completed: 2019-09-20 03:11:42.221146294 +0000 UTC
#9 duration: 6.925038937s
From above, you can see the first time, the build will download pyyaml from internet.
2.
The second execute build:
docker build --progress=plain -t abc:1 . --no-cache
The second log:
#9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install...
#9 digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5
#9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml"
#9 started: 2019-09-20 03:16:58.588157354 +0000 UTC
#9 1.786 Collecting pyyaml
#9 2.234 Installing collected packages: pyyaml
#9 2.270 Successfully installed pyyaml-5.1.2
#9completed: 2019-09-20 03:17:01.933398002 +0000 UTC
#9 duration: 3.345240648s
From above, you can see the build no longer download package from internet, just use the cache. NOTE, this is not the traditional docker build cache as I have use --no-cache, it's /root/.cache/pip which I mount into build.
3.
The third execute build which delete buildkit cache:
docker builder prune
docker build --progress=plain -t abc:1 . --no-cache
The third log:
#9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install...
#9 digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5
#9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml"
#9 started: 2019-09-20 03:19:07.434792944 +0000 UTC
#9 1.894 Collecting pyyaml
#9 2.740 Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)
#9 3.319 Building wheels for collected packages: pyyaml
#9 3.319 Building wheel for pyyaml (setup.py): started
#9 3.560 Building wheel for pyyaml (setup.py): finished with status 'done'
#9 3.560 Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=cea5bc4689e231df7915c2fc3abca225d4ee2e869a7540682aacb6d42eb17053
#9 3.560 Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030
#9 3.580 Successfully built pyyaml
#9 3.585 Installing collected packages: pyyaml
#9 3.622 Successfully installed pyyaml-5.1.2
#9completed: 2019-09-20 03:19:12.530742712 +0000 UTC
#9 duration: 5.095949768s
From above, you can see if delete buildkit cache, the package download again.
In a word, it will give you a shared cache between several times build, and this cache will only be mounted when image build. But, the image self will not have these cache, so avoid a lots of intermediate layer in image.
这篇关于在Docker构建中使用pip缓存目录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
