如何在不同的机器上重用已编译的源代码 [英] How to re-use compiled sources in different machines

问题描述

为了加快我们的开发工作流程，我们拆分测试并在多个代理上并行运行每个部分.但是，编译测试源似乎要花费大部分时间来执行测试步骤.

To speed up our development workflow we split the tests and run each part on multiple agents in parallel. However, compiling test sources seem to take most of the time for the testing steps.

为了避免这种情况，我们使用 sbt test:compile 预编译测试，并使用编译的目标构建一个 docker 镜像.

To avoid this, we pre-compile the tests using sbt test:compile and build a docker image with compiled targets.

稍后，此图像在每个代理中用于运行测试.但是，即使存在已编译的类，它似乎还是要重新编译测试和应用程序源.

Later, this image is used in each agent to run the tests. However, it seems to recompile the tests and application sources even though the compiled classes exists.

有没有办法让 sbt 使用现有的编译目标?

Is there a way to make sbt use existing compiled targets?

更新:提供更多上下文

这个问题与 scala 和 sbt(因此是 sbt 标签)严格相关.

The question strictly relates to scala and sbt (hence the sbt tag).

我们的 CI 流程分为多个阶段.它大致是这样的.

Our CI process is broken down in to multiple phases. Its roughly something like this.

• 第一阶段:使用sbt compile使用SBT将Scala项目编译成java bitcode我们在同一个测试中使用sbt test:compile编译测试源代码目标捆绑在 docker 镜像中并推送到远程存储库，

• stage 1: Use SBT to compile Scala project into java bitecode using sbt compile We compile the test sources in the same test using sbt test:compile The targes are bundled in a docker image and pushed to the remote repository,

阶段 2:我们使用多个代理来并行拆分和运行测试.测试从构建的 docker 镜像运行，因此环境是相同的.但是，运行 sbt test 会导致项目重新编译，即使通过编译的bitcode存在.

stage 2: We use multiple agents to split and run tests in parallel. The tests run from the built docker image, so the environment is the same. However, running sbt test causes the project to recompile even through the compiled bitecode exists.

为了说明这一点，我基本上想在一台机器上编译并在另一台机器上运行编译好的测试源，而无需重新编译

To make this clear, I basically want to compile on one machine and run the compiled test sources in another without re-compiling

更新

我不认为 https://stackoverflow.com/a/37440714/8261 是同样的问题，因为与它不同的是，我不挂载卷或在主机上构建.一切都在 docker 中编译和运行，但分为两个构建阶段.因此文件修改时间和路径保持不变.

I don't think https://stackoverflow.com/a/37440714/8261 is the same problem because unlike it, I don't mount volumes or build on the host machine. Everything is compiled and run within docker but in two build stages. The file modified times and paths are retained the same because of this.

调试输出是这样的

Initial source changes: 
    removed:Set()
    added: Set()
    modified: Set()
Invalidated products: Set(/app/target/scala-2.12/classes/Class1.class, /app/target/scala-2.12/classes/graph/Class2.class, ...)
External API changes: API Changes: Set()
Modified binary dependencies: Set()
Initial directly invalidated classes: Set()

Sources indirectly invalidated by:
    product: Set(/app/Class4.scala, /app/Class5.scala, ...)
    binary dep: Set()
    external source: Set()
All initially invalidated classes: Set()
All initially invalidated sources:Set(/app/Class4.scala, /app/Class5.scala, ...)
Recompiling all 304 sources: invalidated sources (266) exceeded 50.0% of all sources
Compiling 302 Scala sources and 2 Java sources to /app/target/scala-2.12/classes ...

它没有初始源更改，但产品已失效.

It has no Initial source changes, but products are invalidated.

更新:要重现的最小项目

我创建了一个最小的 sbt 项目来重现该问题.https://github.com/pulasthibandara/sbt-docker-recomplile

I created a minimal sbt project to reproduce the issue. https://github.com/pulasthibandara/sbt-docker-recomplile

如您所见，除了在新步骤(新容器)的第二阶段运行之外，构建阶段之间没有任何变化.

As you can see, nothing changes between the build stages, other than running in the second stage in a new step (new container).

推荐答案

虽然 https://stackoverflow.com/a/37440714/8261指出了正确的方向，根本问题和解决方案是不同的.

While https://stackoverflow.com/a/37440714/8261 pointed at the right direction, the underlying issue and the solution for this was different.

问题

SBT 在 docker 构建的不同阶段运行时似乎会重新编译所有内容.这是因为 docker 会压缩在每个阶段创建的图像，从而从源中去除 lastModifiedDate 的毫秒部分.

SBT seems to recompile everything when it's run on different stages of a docker build. This is because docker compresses images created in each stage, which strips out the millisecond portion of the lastModifiedDate from sources.

SBT 在确定源是否已更改时取决于 lastModifiedDate，并且由于其不同(毫秒部分)，构建会触发完整的重新编译.

SBT depends on lastModifiedDate when determining if sources have changed, and since its different (the milliseconds part) the build triggers a full recompilation.

解决方案

• Java 8:按照 https://github.com/sbt/sbt/issues/4168#issuecomment-417655678 解决此问题.

• Java 8: Setting -Dsbt.io.jdktimestamps=true when running SBT as recommended in https://github.com/sbt/sbt/issues/4168#issuecomment-417655678 to workaround this issue.

更新:遵循 https://github.com/sbt/sbt/issues/4168#中的推荐issuecomment-417658294

我通过在 docker 文件中设置 SBT_OPTS 环境变量来解决这个问题

I solved the issue by setting SBT_OPTS env variable in the docker file like

ENV SBT_OPTS="${SBT_OPTS} -Dsbt.io.jdktimestamps=true"

测试项目已使用此变通方法进行更新.

The test project has been updated with this workaround.

这篇关于如何在不同的机器上重用已编译的源代码的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！