如何防止gitlab ci每次都下载sbt? [英] how to prevent gitlab ci from downloading sbt every time?
问题描述
我们有一个使用 gitlab ci 构建的 play2/scala
应用程序.
We have a play2/scala
application which we are building with gitlab ci.
我们的.gitlab-ci.yml
(至少是重要的部分)如下所示:
Our .gitlab-ci.yml
(at least the important part) looks as follows:
image: hseeberger/scala-sbt
variables:
SBT_GLOBAL_BASE_DIR: "$CI_PROJECT_DIR/cache/.sbt"
IVY2_CACHE_DIR: "$CI_PROJECT_DIR/cache/.ivy2"
SBT_BOOT_DIR: "$CI_PROJECT_DIR/cache/.sbt/boot"
M2_HOME_DIR: "$CI_PROJECT_DIR/cache/.m2"
before_script:
# Log the sbt version
- sbt sbt-version
build:
stage: build
script:
- ./build.sh
使用 build.sh
:
sbt -Dsbt.global.base=$SBT_GLOBAL_BASE_DIR
-Dsbt.ivy.home=$IVY2_CACHE_DIR
-Dsbt.boot.directory=$SBT_BOOT_DIR
compile
不幸的是,我们的管道在所有步骤(构建、验证、部署)中始终运行大约 30-40 分钟.大部分时间都花在了一遍又一遍地下载 sbt
上,真的很烦人.
Unfortunately, our pipeline always runs for around 30-40 minutes with all the steps (build, verification, deploy). Most of the time it spends by downloading sbt
over and over again what is really annoying.
我可能对 gitlab ci runners
了解不够,但据我了解,通过使用 hseeberger/scala-sbt 作为镜像,sbt
应该是全局可用的,应该不需要下载.
I might not know enough about gitlab ci runners
but from my understand, by using hseeberger/scala-sbt as the image, sbt
should be globally available and there should be no need to download it.
然后 这个 来自 gitlab 的解决方案是不必要的.
Then also this solution from gitlab would not be necessary.
无论如何,如果服务器运行任何 sbt
命令时,每次部署期间不会总共下载 6 次 sbt
,我会很高兴.
Anyhow, I would be glad if sbt
would not be downloaded totally 6 times during each deployment whenever the server runs any sbt
command.
谁能解释我如何以正确的方式使用正确的 image
或 image
或者如何缓存 sbt
的东西?
Can someone explain me how to use the right image
or the image
in the right way or otherwise how I can cache the sbt
stuff?
在过去的几天里,我与 docker
和 gitlab ci
进行了很多斗争.我发现这个问题与 不要下载互联网.拥有所有依赖项似乎是一项艰巨的任务,最好通过挂载它们来完成.不幸的是,这在共享的 gitlab ci 运行器上是不可能的.
Over the last days I fought a lot with docker
and gitlab ci
. I found that this problems is pretty much the same as described in don't downloading the internet. It seems that this is some hard task to have all the dependencies and should be best done by mounting them. That's unfortunately not possible as such on a shared gitlab ci runner.
我继续发现 sbt-docker 它允许您从 sbt-docker代码>build.sbt 文件.使用 包基本方法 我试过将项目的所有本地可用依赖项作为 全局 sbt 插件.但这也无济于事.
I went on and discovered sbt-docker which allows you to build docker containers from a build.sbt
file. With the package basic approach I tried to include all the locally available dependencies for the project into the container as global sbt plugins. But also this didn't help.
我最后的发现是关于 maven
解决方案 并尝试将其转换为我们的 sbt
项目:
My last discovery was this answer regarding the maven
solution and tried to translate that into our sbt
project:
image: hseeberger/scala-sbt
variables:
MAVEN_OPTS: -Dmaven.repo.local=/cache/maven.repository
stages:
- build
- test
- staging
- deploy
build:
stage: build
script:
- sbt compile -Dsbt.ivy.home=/cache/.ivy2 -Dsbt.global.base=/cache/.sbt/0.13 -Dsbt.boot.directory=/cache/.sbt/boot -Dsbt.repository.config=/cache/.sbt/repositories
我可以再次访问 gitlab ci
日志.它们基本上如下所示:
I can access the gitlab ci
logs again. They look basically as follows:
[info] Loading project definition from /builds/kwiqjobs/backend/project
[info] Updating {file:/builds/kwiqjobs/backend/project/}backend-build...
[info] Resolving com.typesafe.play#sbt-plugin;2.5.4 ...
[info] Resolving com.typesafe.play#sbt-plugin;2.5.4 ...
[info] Resolving com.typesafe.play#sbt-routes-compiler_2.10;2.5.4 ...
[info] Resolving com.typesafe.play#sbt-routes-compiler_2.10;2.5.4 ...
[info] Resolving org.scala-lang#scala-library;2.10.6 ...
[info] Resolving com.typesafe.play#twirl-api_2.10;1.1.1 ...
[info] Resolving com.typesafe.play#twirl-api_2.10;1.1.1 ...
... a **lot** more
[info] [SUCCESSFUL ] com.typesafe.sbt#sbt-twirl;1.1.1!sbt-twirl.jar (1033ms)
[info] downloading https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-native-packager/scala_2.10/sbt_0.13/1.0.3/jars/sbt-native-packager.jar ...
[info] [SUCCESSFUL ] com.typesafe.sbt#sbt-native-packager;1.0.3!sbt-native-packager.jar (954ms)
[info] downloading https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-web/scala_2.10/sbt_0.13/1.3.0/jars/sbt-web.jar ...
[info] [SUCCESSFUL ] com.typesafe.sbt#sbt-web;1.3.0!sbt-web.jar (1010ms)
[info] downloading https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-js-engine/scala_2.10/sbt_0.13/1.1.3/jars/sbt-js-engine.jar ...
[info] [SUCCESSFUL ] com.typesafe.sbt#sbt-js-engine;1.1.3!sbt-js-engine.jar (1147ms)
[info] downloading https://repo1.maven.org/maven2/com/typesafe/play/twirl-api_2.10/1.1.1/twirl-api_2.10-1.1.1.jar ...
[info] [SUCCESSFUL ] com.typesafe.play#twirl-api_2.10;1.1.1!twirl-api_2.10.jar (89ms)
[info] downloading https://repo1.maven.org/maven2/commons-io/commons-io/2.4/commons-io-2.4.jar ...
[info] [SUCCESSFUL ] commons-io#commons-io;2.4!commons-io.jar (48ms)
a **lot** more
[info] Done updating.
[info] Compiling 228 Scala sources and 4 Java sources to /builds/kwiqjobs/backend/target/scala-2.11/classes...
[info] 'compiler-interface' not yet compiled for Scala 2.11.8. Compiling...
[info] Compilation completed in 17.735 s
[success] Total time: 149 s, completed Jan 20, 2017 2:22:52 PM
Build succeeded
我想摆脱所有的下载.
推荐答案
如果你不想使用自定义图像,最好的解决方案是使用 Gitlab CI 的 缓存机制.
If you don't want to use custom made images, the best solution is to use Gitlab CI's caching mechanism.
要做到这一点有点困难,但是 这篇博文描述了如何为 SBT 做到这一点.
It's a little hard to get it right, but this blog post describes how to do it for SBT.
引自博文,小错误自行更正:
Quoted from the blog post, minor errors corrected by myself:
# some parts originally from https://github.com/randm-ch/units-of-information/blob/master/.gitlab-ci.yml
image: "hseeberger/scala-sbt"
variables:
SBT_VERSION: "0.13.9"
SBT_OPTS: "-Dsbt.global.base=sbt-cache/.sbtboot -Dsbt.boot.directory=sbt-cache/.boot -Dsbt.ivy.home=sbt-cache/.ivy"
cache:
key: "$CI_BUILD_REF_NAME" # contains either the branch or the tag, so it's caching per branch
untracked: true
paths:
- "sbt-cache/.ivy/cache"
- "sbt-cache/.boot"
- "sbt-cache/.sbtboot"
- "sbt-cache/target"
stages:
- test
test:
script:
- sbt test
第二个例子,也包括apt-get
缓存
这是我在项目中使用的,可用于更一般的用例和 Docker 映像:
Second example, also including apt-get
caching
This is what I used for my project, usable for more general use cases and Docker images:
image: java:8
stages:
- test
variables:
SBT_VERSION: "0.13.9"
SBT_OPTS: "-Dsbt.global.base=sbt-cache/.sbtboot -Dsbt.boot.directory=sbt-cache/.boot -Dsbt.ivy.home=sbt-cache/.ivy"
SBT_CACHE_DIR: "sbt-cache/.ivy/cache"
cache:
key: "$CI_BUILD_REF_NAME" # contains either the branch or the tag, so it's caching per branch
untracked: true
paths:
- "apt-cache/"
- "sbt-cache/.ivy/cache"
- "sbt-cache/.boot"
- "sbt-cache/.sbtboot"
- "sbt-cache/target"
before_script:
- export APT_CACHE_DIR=`pwd`/apt-cache
- mkdir -pv $APT_CACHE_DIR
- ls $APT_CACHE_DIR || echo "no apt-cache dir found"
- apt-get -o dir::cache::archives=$APT_CACHE_DIR update -y
- apt-get -o dir::cache::archives=$APT_CACHE_DIR install apt-transport-https -y
# Install SBT
- mkdir -pv $SBT_CACHE_DIR
- ls $SBT_CACHE_DIR || echo "no ivy2 cache fir found"
- echo "deb http://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list
- apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
- apt-get -o dir::cache::archives=$APT_CACHE_DIR update -y
- apt-get -o dir::cache::archives=$APT_CACHE_DIR install sbt -y
- sbt -v sbtVersion
test:
stage: test
script:
- sbt -v sbtVersion
这篇关于如何防止gitlab ci每次都下载sbt?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!