如何在Dockerfile中添加大型HTTP文件并将其从图像层中排除? [英] How do I add big HTTP files in a Dockerfile and exclude them from image layers?

问题描述

我们的 Nexus服务器为我们的Java项目(包括安装程序)提供了构建工件.该安装程序确实很大(> 1GB).我想在Dockerfile中检索并使用它.

Our Nexus server provides build artifacts for our Java project including its installer. That installer is really big (>1GB). I would like to retrieve and use it in a Dockerfile.

到目前为止，我所做的是:

What I did so far is the following:

FROM debian:jessie
...
RUN apt-get install -y curl xmllib-xpath-perl
ENV PROJECT_VERSION x.y.z-SNAPSHOT
...
RUN VERSION=`curl --silent "http://nexus:8081/service/local/artifact/maven/resolve?r=public&g=my.group.id&a=installer&v=${PROJECT_VERSION}&e=sh&c=linux64" | xpath -q -s '' -e '//data/version/text()'` \
    && echo Version:\'${VERSION}\' \
    && curl --silent http://nexus/content/groups/public/my/group/id/installer/${PROJECT_VERSION}/installer-${VERSION}-linux64.sh \
        --create-dirs \
        --output ${INSTALL_DIR}/installer.sh \
    && sh ${INSTALL_DIR}/installer.sh <someArgs> \
    && rm ${INSTALL_DIR}/installer.sh
...

使用这种方法，我能够:

With that approach I am able to:

• 查询Nexus，以提供所提供的${PROJECT_VERSION}的最新SNAPSHOT版本，该版本在docker build
期间注销
• 使用该版本下载相应的安装程序二进制文件
• 执行安装程序二进制文件
• 执行后立即将安装程序二进制文件删除，以不将其存储在创建的Docker映像层中

• Query Nexus to provide the latest SNAPSHOT version for the provided ${PROJECT_VERSION} which is logged out during docker build
• Use that version to download the corresponding installer binary
• Execute the installer binary
• Delete the installer binary immediately after execution to not have it stored within the created Docker image layer

缺少的内容:

• 每当将新安装程序部署到Nexus时，我都必须使用docker build --no-cache构建Docker映像.否则，Docker将无法使缓存无效，并重新运行同时已部署到Nexus的较新安装程序的安装步骤.

• Whenever a new installer gets deployed to Nexus I have to build the Docker image with docker build --no-cache. Otherwise Docker is not able to invalidate its cache and re-run the installation step for a newer installer that was meanwhile deployed to Nexus.

因此，根据文档，我使用ADD语句尝试了另一种方法，因为它们具有缓存功能.但这不起作用，因为我需要为ADD语句提供一个参数，该语句是由上一步查询Nexus以获取正确的SNAPSHOT版本的步骤设置的:

So I tried a different approach using the ADD statement as those have caching capabilities according to the documentation. But that does not work since I need to provide a parameter to the ADD statement that is set by a previous step querying Nexus for the correct SNAPSHOT version:

FROM debian:jessie
...
RUN apt-get install -y curl xmllib-xpath-perl
ENV PROJECT_VERSION x.y.z-SNAPSHOT
...
ADD http://nexus:8081/service/local/artifact/maven/resolve?r=public&g=my.group.id&a=installer&v=${PROJECT_VERSION}&e=sh&c=linux64 ${INSTALL_DIR}/version.xml
RUN cat ${INSTALL_DIR}/version.xml | xpath -q -s '' -e '//data/version/text()' > ${INSTALL_DIR}/version.txt

# FIXME: Somehow do a `cat ${INSTALL_DIR}/version.txt to set the ENV ${VERSION} variable ?!

ADD http://nexus/content/groups/public/my/group/id/installer/${PROJECT_VERSION}/installer-${VERSION}-linux64.sh ${INSTALL_DIR}/installer.sh
RUN ${INSTALL_DIR}/installer.sh <someArgs> && rm ${INSTALL_DIR}/installer.sh
...

该方法行不通，因为:

• 无法将Dockerfile中的${VERSION}环境变量设置为存储在version.txt文件中的版本.
• 无法阻止将安装程序存储在映像层中.

• It is not possible to set the ${VERSION} environment variable within the Dockerfile to the version stored within the version.txt file.
• It is not possible to prevent having the installer stored within an image layer.

但是至少这将使用适当的缓存来将现有映像层重新用于旧的安装程序版本，并在Nexus上部署新的安装程序版本时创建新的映像层.

But at least this would use proper caching to re-use existing image layers for old installer versions and create new ones whenever a new installer version on Nexus gets deployed.

所以问题是:如何同时从Docker映像层中启用适当的缓存，缓存无效以及大安装程序文件的排除?

So the question is: How do I enable proper caching, cache invalidation and exclusion of the big installer file from the Docker image layers at the same time?

编辑:我找到了一种方法，可以通过使用其他Nexus API来使图像图层的缓存正常工作:

EDIT: I found a way to get the caching of image layers working properly by using an other Nexus API:

FROM debian:jessie
...
ENV PROJECT_VERSION x.y.z-SNAPSHOT
...
ADD http://nexus:8081/service/local/artifact/maven/content?r=public&g=my.group.id&a=installer&v=${PROJECT_VERSION}&e=sh&c=linux64 ${INSTALL_DIR}/installer.sh
RUN sh ${INSTALL_DIR}/installer.sh <someArgs> \
    && rm ${INSTALL_DIR}/installer.sh
...

但是仍然存在在图像层中包含很大的安装程序文件的问题，因为在该代码段中使用了ADD机制.

But still the problem of having a very big installer file included in the image layers remains since in that code snipped the ADD mechanism is used.

关于如何从ADD语句提供的缓存及其正确无效中受益的任何想法，但同时不将添加的文件包括在图像历史记录中?

Any ideas about how to benefit from the caching and its correct invalidation provided by the ADD statement but at the same time not include the added file into the images history?

推荐答案

如何在一个长期运行的命令中执行curl/wget，安装和删除?

How about doing curl/wget, install and remove in one long run command?

更新，结合较小资源的ADD，请参阅TC的详细答案.

Update in combination with ADD of a smaller resource, see TC's detailed answer.

这篇关于如何在Dockerfile中添加大型HTTP文件并将其从图像层中排除?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！