如何在Docker中缓存Maven依赖项 [英] How to cache maven dependencies in Docker

问题描述

我正在处理一个约200MB依赖项的项目，由于带宽有限，我想避免无用的上传.

I'm working on a project with ~200MB dependencies and i'd like to avoid useless uploads due to my limited bandwidth.

当我推送Dockerfile时(稍后会附加它)，即使我不碰pom.xml，我也总是上传〜200MB:

When I push my Dockerfile (i'll attach it in a moment), I always have a ~200MB upload even if I didn't touch the pom.xml:

FROM maven:3.6.0-jdk-8-slim

WORKDIR /app

ADD pom.xml /app

RUN mvn verify clean --fail-never

COPY ./src /app/src

RUN mvn package

ENV CONFIG_FOLDER=/app/config
ENV DATA_FOLDER=/app/data
ENV GOLDENS_FOLDER=/app/goldens
ENV DEBUG_FOLDER=/app/debug

WORKDIR target

CMD ["java","-jar","-Dlogs=/app/logs", "myProject.jar"]

此Dockerfile应该制作一个200MB的fatJAR，其中包括所有依赖项，这就是为什么每次都会发生〜200MB的上载.我想要实现的是构建一个包含所有依赖项的图层，并打包"到打包阶段，以不将依赖项JAR包含在fatJAR中，而是在给定目录中搜​​索它们.

This Dockerfile should make a 200MB fatJAR including all the dependencies, that's why the ~200MB upload that occurs everytime. What i would like to achieve is building a Layer with all the dependencies and "tell" to the packaging phase to not include the dependencies JARs into the fatJAR but to search for them inside a given directory.

我想知道如何在构建过程之前构建一个执行mvn dependency:copy-dependencies的脚本，然后将目录复制到容器中；然后构建一个非胖" JAR，将所有那些依赖项都链接在一起，而实际上并没有将其复制到其中.

I was wondering to build a script that executes mvn dependency:copy-dependencies before the building process and then copying the directory to the container; then building a "non-fat"JAR that has all those dependencies only linked and not actually copied into it.

这可能吗?

我发现容器的Maven本地存储库位于/root/.m2下.所以我结束了一个非常简单的脚本，如下所示:

I discovered that the Maven Local Repository of the container is located under /root/.m2. So I ended making a very simple script like this:

BuildDocker.sh

mvn verify -clean --fail-never
mv ~/.m2 ~/git/myProjectRepo/.m2

sudo docker build -t myName/myProject:"$1"

并按如下所示编辑 Dockerfile :

# Use an official Python runtime as a parent image
FROM maven:3.6.0-jdk-8-slim

# Copy my Mavne Local Repository into the container thus creating a new layer
COPY ./.m2 /root/.m2

# Set the working directory to /app
WORKDIR /app

# Copy the pom.xml
ADD pom.xml /app

# Resolve and Download all dependencies: this will be done only if the pom.xml has any changes
RUN mvn verify clean --fail-never

# Copy source code and configs 
COPY ./src /app/src

# create a ThinJAR
RUN mvn package


# Run the jar
...

在构建过程后，我说/root/.m2拥有我所有的目录，但是一旦启动JAR，我就会得到:

After the building process i stated that /root/.m2 has all the directories I but as soon as i launch the JAR i get:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Priority
    at myProject.ThreeMeans.calculate(ThreeMeans.java:17)
    at myProject.ClusteringStartup.main(ClusteringStartup.java:7)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Priority
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 2 more

也许我不应该通过java -jar运行它?

Maybe i shouldn't run it through java -jar?

推荐答案

如果我正确理解了您想要实现的目标，那么问题是要避免创建一个包含所有Maven依赖项的 fat jar 每个Docker构建(以减轻重建后要推送的Docker层的大小).

If I understand correctly what you'd like to achieve, the problem is to avoid creating a fat jar with all Maven dependencies at each Docker build (to alleviate the size of the Docker layers to be pushed after a rebuild).

如果是，您可能对 Spring Boot Thin Launcher ，它也适用于非Spring-Boot项目.相应的GitHub存储库的README.md中提供了一些全面的文档: https://github.com/dsyer/spring-boot-thin-launcher#自述文件

If yes, you may be interested in the Spring Boot Thin Launcher, which is also applicable for non-Spring-Boot projects. Some comprehensive documentation is available in the README.md of the corresponding GitHub repo: https://github.com/dsyer/spring-boot-thin-launcher#readme

总而言之，在您的pom.xml中添加以下插件声明就足够了:

To sum up, it should suffice to add the following plugin declaration in your pom.xml:

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <!--<version>${spring-boot.version}</version>-->
            <dependencies>
                <dependency>
                    <groupId>org.springframework.boot.experimental</groupId>
                    <artifactId>spring-boot-thin-layout</artifactId>
                    <version>1.0.19.RELEASE</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>

理想情况下，此解决方案应与标准Dockerfile设置结合使用，以受益于Docker的缓存(请参阅下面的典型示例).

Ideally, this solution should be combined with a standard Dockerfile setup to benefit from Docker's cache (see below for a typical example).

以下参考文献中给出了Dockerfile的原型，该原型避免了每次仅接触源代码文件(src/*)时重新下载所有Maven依赖项的情况:
https://whitfin.io/speeding-up-maven-docker-builds/

The archetype of a Dockerfile that avoids re-downloading all Maven dependencies at each build if only source code files (src/*) have been touched is given in the following reference:
https://whitfin.io/speeding-up-maven-docker-builds/

更准确地说，建议的Dockerfile如下:

To be more precise, the proposed Dockerfile is as follows:

# our base build image
FROM maven:3.5-jdk-8 as maven

WORKDIR /app

# copy the Project Object Model file
COPY ./pom.xml ./pom.xml

# fetch all dependencies
RUN mvn dependency:go-offline -B

# copy your other files
COPY ./src ./src

# build for release
# NOTE: my-project-* should be replaced with the proper prefix
RUN mvn package && cp target/my-project-*.jar app.jar


# smaller, final base image
FROM openjdk:8u171-jre-alpine
# OPTIONAL: copy dependencies so the thin jar won't need to re-download them
# COPY --from=maven /root/.m2 /root/.m2

# set deployment directory
WORKDIR /app

# copy over the built artifact from the maven image
COPY --from=maven /app/app.jar ./app.jar

# set the startup command to run your binary
CMD ["java", "-jar", "/app/app.jar"]

请注意，它依赖于所谓的 multi- Docker的阶段构建功能 (存在两个FROM指令)，这意味着最终映像将比maven基本映像本身小很多.
(如果在开发阶段对该功能不感兴趣，则可以删除FROM openjdk:8u171-jre-alpine和COPY --from=maven /app/app.jar ./app.jar行.)

Note that it relies on the so-called multi-stage build feature of Docker (presence of two FROM directives), implying the final image will be much smaller than the maven base image itself.
(If you are not interested in that feature during the development phase, you can remove the lines FROM openjdk:8u171-jre-alpine and COPY --from=maven /app/app.jar ./app.jar.)

在这种方法中，使用RUN mvn dependency:go-offline -B 在COPY ./src ./src行之前获取Maven依赖项(以受益于Docker的缓存).

In this approach, the Maven dependencies are fetched with RUN mvn dependency:go-offline -B before the line COPY ./src ./src (to benefit from Docker's cache).

但是请注意，dependency:go-offline标准目标并非完美"，因为一些动态依赖项/插件仍可能会在mvn package步骤中触发某些重新下载. 如果这对您来说是个问题(例如，如果您确实想离线工作)，则可以看看其他 SO答案，建议使用提供de.qaware.maven:go-offline-maven-plugin:resolve-dependencies目标的专用插件.

Note however that the dependency:go-offline standard goal is not "perfect" as a few dynamic dependencies/plugins may still trigger some re-downloading at the mvn package step. If this is an issue for you (e.g. if at some point you'd really want to work offline), you could take at look at that other SO answer that suggests using a dedicated plugin that provides the de.qaware.maven:go-offline-maven-plugin:resolve-dependencies goal.

这篇关于如何在Docker中缓存Maven依赖项的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！