为什么此COPY命令会使我的Docker缓存无效? [英] Why does my Docker cache get invalidated by this COPY command?

问题描述

我的CI系统中的docker构建器在不活动后被破坏，从而丢失了本地缓存.我正在使用-cache-from ，方法是先从quay.io存储库中提取最新图像，然后在下一个版本中将其用作-cache-from .我正在运行docker版本 17.12.0-ce .Dockerfile(相关部分)如下所示:

My docker builders in CI system get destroyed after inactivity, thus losing the local cache. I am using --cache-from by first pulling the most recent image from quay.io repo and then using that as a --cache-from in the next build. I am running docker version 17.12.0-ce. Dockerfile (for the pertinent part) looks like:

FROM ubuntu:16.04
RUN apt-get update && apt-get install -y \
ant \
build-essential \
software-properties-common \
libncurses5-dev \
libncursesw5-dev \
libcurl4-openssl-dev \
libboost-dev \
libfreetype6-dev \
zlib1g-dev \
r-base \
default-jdk \
python-dev \
python-setuptools \
python-pip \
python3-dev \
python3-setuptools \
python3-pip \
git \
wget \
unzip \
ghostscript \
pkg-config


RUN mkdir /software
WORKDIR /software
ENV PATH="/software:${PATH}"

RUN git clone --branch v0.2.19 --single-branch 
https://github.com/xianyi/OpenBLAS
RUN cd OpenBLAS && make FC=gfortran TARGET=NEHALEM USE_THREAD=0 && make 
PREFIX=/opt/openblas install
ENV LD_LIBRARY_PATH="/opt/openblas/lib:${LD_LIBRARY_PATH}"

# Install samtools dependencies
RUN wget http://zlib.net/zlib-1.2.11.tar.gz && tar -xvf zlib-1.2.11.tar.gz
RUN cd zlib-1.2.11 && ./configure && make && make install
RUN wget http://bzip.org/1.0.6/bzip2-1.0.6.tar.gz && tar -xvf bzip2-
1.0.6.tar.gz
RUN cd bzip2-1.0.6 && make && make install
RUN wget https://tukaani.org/xz/xz-5.2.3.tar.gz && tar -xvf xz-5.2.3.tar.gz
RUN cd xz-5.2.3 && ./configure && make && make install

RUN pip install common python-dateutil cython

RUN pip3 install common python-dateutil cython

# Install numpy 1.11.3 (python2/3)
RUN git clone --branch v1.11.3 --single-branch https://github.com/numpy/numpy
COPY /docker_image/site.cfg numpy/
RUN cd numpy && python setup.py install
RUN cd numpy && python3 setup.py install

当我使用(干净的机器，没有缓存)运行构建时:

When I run my build with (clean machine, with nothing in cache):

docker pull quay.io/myorganization/myimage:tag

，然后使用

docker build --cache-from=quay.io/myorganization/myimage:tag -f docker_image/Dockerfile -t quay.io/myorganization/myimage:newtag .

构建使用缓存，直到 COPY/docker_image/site.cfg numpy/使缓存无效.我的.dockerignore看起来像:

the build uses cache until COPY /docker_image/site.cfg numpy/ invalidates the cache. My .dockerignore looks like:

.git*

所以事情发生变化应该不会有问题.如果我不小心忽略了一些重要信息，请询问，我会立即提供.对于可能导致此特定位置的缓存失效的任何想法，将受到高度赞赏.

so things changing there should not be the problem. If I accidentally omitted some important information needed, please ask and I will promptly provide that. Any ideas on what could cause the cache invalidation on this particular spot would be highly appreciated.

edit:通过执行以下操作，即使我在两次构建之间的存储库中未进行任何更改，也会发生此缓存无效:使用tag1的构建图像，Push image到quay.io，然后在干净的计算机上克隆git repo，拉取图像(tag1)，使用tag2构建图像.可能会改变numpy回购元数据吗?(请注意:据我所知，--single-branch不应提取有关该存储库中其他分支的任何信息.)

edit: This cache invalidation happens even if I don't change anything in the repo between builds, by doing the following: Build image using tag1, Push image to quay.io, then on a clean machine Clone git repo, Pull image (tag1), Build image with tag2. Could it be something that changes in the numpy repo metadata? (Note: --single-branch should not, to my understanding, pull any info about other branches in that repo).

推荐答案

COPY 或 ADD 命令的docker缓存使用文件和目录的哈希值.该哈希中包括每个文件的内容，甚至是文件的权限.因此，如果其中任何一个更改了一个字节，哈希将有所不同，并且docker将发生缓存未命中，从而迫使该行重新运行.

The docker cache for a COPY or ADD command uses a hash of the files and directories. Included in that hash are the contents of every file, and even the permissions on the files. So if any of these changed by a single byte, the hash will be different and docker will have a cache miss, forcing the line to be rerun.

从第一个高速缓存未命中开始，所有剩余的行都需要重建，因为上一层现在是新的，并且尚未用于执行以下任何步骤.

From the point of the first cache miss, all remaining lines will need to be rebuilt since the preceding layer is now new and has not be used to run any of the following steps.

这篇关于为什么此COPY命令会使我的Docker缓存无效?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！