如何防止gitlab ci每次都下载sbt? [英] how to prevent gitlab ci from downloading sbt every time?

查看：43
本文介绍了如何防止gitlab ci每次都下载sbt?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我们有一个使用 gitlab ci 构建的 play2/scala 应用程序.

We have a play2/scala application which we are building with gitlab ci.

我们的.gitlab-ci.yml(至少是重要的部分)如下所示:

Our .gitlab-ci.yml (at least the important part) looks as follows:

image: hseeberger/scala-sbt

variables:
  SBT_GLOBAL_BASE_DIR: "$CI_PROJECT_DIR/cache/.sbt"
  IVY2_CACHE_DIR: "$CI_PROJECT_DIR/cache/.ivy2"
  SBT_BOOT_DIR:  "$CI_PROJECT_DIR/cache/.sbt/boot"
  M2_HOME_DIR: "$CI_PROJECT_DIR/cache/.m2"

before_script:
  # Log the sbt version
  - sbt sbt-version

build:
  stage: build
  script:
    - ./build.sh

使用 build.sh:

sbt -Dsbt.global.base=$SBT_GLOBAL_BASE_DIR 
  -Dsbt.ivy.home=$IVY2_CACHE_DIR 
  -Dsbt.boot.directory=$SBT_BOOT_DIR 
  compile

不幸的是，我们的管道在所有步骤(构建、验证、部署)中始终运行大约 30-40 分钟.大部分时间都花在了一遍又一遍地下载 sbt 上，真的很烦人.

Unfortunately, our pipeline always runs for around 30-40 minutes with all the steps (build, verification, deploy). Most of the time it spends by downloading sbt over and over again what is really annoying.

我可能对 gitlab ci runners 了解不够，但据我了解，通过使用 hseeberger/scala-sbt 作为镜像，sbt 应该是全局可用的，应该不需要下载.

I might not know enough about gitlab ci runners but from my understand, by using hseeberger/scala-sbt as the image, sbt should be globally available and there should be no need to download it.

然后 这个 来自 gitlab 的解决方案是不必要的.

Then also this solution from gitlab would not be necessary.

无论如何，如果服务器运行任何 sbt 命令时，每次部署期间不会总共下载 6 次 sbt，我会很高兴.

Anyhow, I would be glad if sbt would not be downloaded totally 6 times during each deployment whenever the server runs any sbt command.

谁能解释我如何以正确的方式使用正确的 image 或 image 或者如何缓存 sbt 的东西?

Can someone explain me how to use the right image or the image in the right way or otherwise how I can cache the sbt stuff?

在过去的几天里，我与 docker 和 gitlab ci 进行了很多斗争.我发现这个问题与 不要下载互联网.拥有所有依赖项似乎是一项艰巨的任务，最好通过挂载它们来完成.不幸的是，这在共享的 gitlab ci 运行器上是不可能的.

Over the last days I fought a lot with docker and gitlab ci. I found that this problems is pretty much the same as described in don't downloading the internet. It seems that this is some hard task to have all the dependencies and should be best done by mounting them. That's unfortunately not possible as such on a shared gitlab ci runner.

我继续发现 sbt-docker 它允许您从 sbt-docker代码>build.sbt 文件.使用 包基本方法 我试过将项目的所有本地可用依赖项作为 全局 sbt 插件.但这也无济于事.

I went on and discovered sbt-docker which allows you to build docker containers from a build.sbt file. With the package basic approach I tried to include all the locally available dependencies for the project into the container as global sbt plugins. But also this didn't help.

我最后的发现是关于 maven 解决方案 并尝试将其转换为我们的 sbt 项目:

My last discovery was this answer regarding the maven solution and tried to translate that into our sbt project:

image: hseeberger/scala-sbt

variables:
  MAVEN_OPTS: -Dmaven.repo.local=/cache/maven.repository

stages:
  - build
  - test
  - staging
  - deploy

build:
  stage: build
  script:
    - sbt compile -Dsbt.ivy.home=/cache/.ivy2 -Dsbt.global.base=/cache/.sbt/0.13 -Dsbt.boot.directory=/cache/.sbt/boot -Dsbt.repository.config=/cache/.sbt/repositories

我可以再次访问 gitlab ci 日志.它们基本上如下所示:

I can access the gitlab ci logs again. They look basically as follows:

[info] Loading project definition from /builds/kwiqjobs/backend/project
[info] Updating {file:/builds/kwiqjobs/backend/project/}backend-build...
[info] Resolving com.typesafe.play#sbt-plugin;2.5.4 ...

[info] Resolving com.typesafe.play#sbt-plugin;2.5.4 ...

[info] Resolving com.typesafe.play#sbt-routes-compiler_2.10;2.5.4 ...

[info] Resolving com.typesafe.play#sbt-routes-compiler_2.10;2.5.4 ...

[info] Resolving org.scala-lang#scala-library;2.10.6 ...

[info] Resolving com.typesafe.play#twirl-api_2.10;1.1.1 ...

[info] Resolving com.typesafe.play#twirl-api_2.10;1.1.1 ...

... a **lot** more

[info]  [SUCCESSFUL ] com.typesafe.sbt#sbt-twirl;1.1.1!sbt-twirl.jar (1033ms)
[info] downloading https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-native-packager/scala_2.10/sbt_0.13/1.0.3/jars/sbt-native-packager.jar ...
[info]  [SUCCESSFUL ] com.typesafe.sbt#sbt-native-packager;1.0.3!sbt-native-packager.jar (954ms)
[info] downloading https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-web/scala_2.10/sbt_0.13/1.3.0/jars/sbt-web.jar ...
[info]  [SUCCESSFUL ] com.typesafe.sbt#sbt-web;1.3.0!sbt-web.jar (1010ms)
[info] downloading https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-js-engine/scala_2.10/sbt_0.13/1.1.3/jars/sbt-js-engine.jar ...
[info]  [SUCCESSFUL ] com.typesafe.sbt#sbt-js-engine;1.1.3!sbt-js-engine.jar (1147ms)
[info] downloading https://repo1.maven.org/maven2/com/typesafe/play/twirl-api_2.10/1.1.1/twirl-api_2.10-1.1.1.jar ...
[info]  [SUCCESSFUL ] com.typesafe.play#twirl-api_2.10;1.1.1!twirl-api_2.10.jar (89ms)
[info] downloading https://repo1.maven.org/maven2/commons-io/commons-io/2.4/commons-io-2.4.jar ...
[info]  [SUCCESSFUL ] commons-io#commons-io;2.4!commons-io.jar (48ms)

a **lot** more
[info] Done updating.
[info] Compiling 228 Scala sources and 4 Java sources to /builds/kwiqjobs/backend/target/scala-2.11/classes...
[info] 'compiler-interface' not yet compiled for Scala 2.11.8. Compiling...
[info]   Compilation completed in 17.735 s
[success] Total time: 149 s, completed Jan 20, 2017 2:22:52 PM
Build succeeded

我想摆脱所有的下载.

推荐答案

如果你不想使用自定义图像，最好的解决方案是使用 Gitlab CI 的 缓存机制.

If you don't want to use custom made images, the best solution is to use Gitlab CI's caching mechanism.

要做到这一点有点困难，但是 这篇博文描述了如何为 SBT 做到这一点.

It's a little hard to get it right, but this blog post describes how to do it for SBT.

引自博文，小错误自行更正:

Quoted from the blog post, minor errors corrected by myself:

# some parts originally from https://github.com/randm-ch/units-of-information/blob/master/.gitlab-ci.yml

image: "hseeberger/scala-sbt"

variables:
  SBT_VERSION: "0.13.9"
  SBT_OPTS: "-Dsbt.global.base=sbt-cache/.sbtboot -Dsbt.boot.directory=sbt-cache/.boot -Dsbt.ivy.home=sbt-cache/.ivy"

cache:
  key: "$CI_BUILD_REF_NAME" # contains either the branch or the tag, so it's caching per branch
  untracked: true
  paths:
    - "sbt-cache/.ivy/cache"
    - "sbt-cache/.boot"
    - "sbt-cache/.sbtboot"
    - "sbt-cache/target"

stages:
  - test

test:
  script:
    - sbt test

第二个例子，也包括apt-get缓存

这是我在项目中使用的，可用于更一般的用例和 Docker 映像:

Second example, also including apt-get caching

This is what I used for my project, usable for more general use cases and Docker images:

image: java:8

stages:
  - test

variables:
  SBT_VERSION: "0.13.9"
  SBT_OPTS: "-Dsbt.global.base=sbt-cache/.sbtboot -Dsbt.boot.directory=sbt-cache/.boot -Dsbt.ivy.home=sbt-cache/.ivy"
  SBT_CACHE_DIR: "sbt-cache/.ivy/cache"

cache:
  key: "$CI_BUILD_REF_NAME" # contains either the branch or the tag, so it's caching per branch
  untracked: true
  paths:
    - "apt-cache/"
    - "sbt-cache/.ivy/cache"
    - "sbt-cache/.boot"
    - "sbt-cache/.sbtboot"
    - "sbt-cache/target"

before_script:
  - export APT_CACHE_DIR=`pwd`/apt-cache
  - mkdir -pv $APT_CACHE_DIR
  - ls $APT_CACHE_DIR || echo "no apt-cache dir found"
  - apt-get -o dir::cache::archives=$APT_CACHE_DIR update -y
  - apt-get -o dir::cache::archives=$APT_CACHE_DIR install apt-transport-https -y
  # Install SBT
  - mkdir -pv $SBT_CACHE_DIR
  - ls $SBT_CACHE_DIR || echo "no ivy2 cache fir found"
  - echo "deb http://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list
  - apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
  - apt-get -o dir::cache::archives=$APT_CACHE_DIR update -y
  - apt-get -o dir::cache::archives=$APT_CACHE_DIR install sbt -y
  - sbt -v sbtVersion

test:
  stage: test
  script:
     - sbt -v sbtVersion

这篇关于如何防止gitlab ci每次都下载sbt?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆