GitLab CI 在构建阶段之间保留环境 [英] GitLab CI preserve environment between build stages

问题描述

我正在开发一个 python 项目并使用 miniconda 来管理我的环境.我正在使用具有以下运行器配置的 GitLab 进行 CI

I am working on a python project and using miniconda to manage my environment. I am using GitLab for CI with the following runner configuration

stages:
  - build
  - test 

build:
  stage: build
  script:
    - if hash $HOME/miniconda/bin/conda 2>/dev/null; 
      then
         export PATH="$HOME/miniconda/bin:$PATH";
      else
        wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh;
        bash miniconda.sh -b -p $HOME/miniconda;
        export PATH="$HOME/miniconda/bin:$PATH";
      fi
    - conda update --yes conda

test:
  stage: test
  script:
    - conda env create --quiet --force --file environment.yml
    - source activate myenv
    - nosetests --with-coverage --cover-erase --cover-package=mypackage --cover-html
    - pylint --reports=n tests/test_final.py
    - pep8 tests/test_final.py
    - grep pc_cov cover/index.html | egrep -o "[0-9]+\%" | awk '{ print "covered " $1;}'

我假设(错误地)我的 build 阶段会设置正确的环境，我可以在其中运行我的 test 阶段.查看这个问题和这个 GitLab 问题 我明白了

I assumed (incorrectly) that my build stage would setup the correct environment in which I could run my test stage. Looking at this question and this GitLab issue I see that

.gitlab-ci.yml 中定义的每个作业都作为单独的构建运行(我们假设没有历史)

each job defined in .gitlab-ci.yml is run as separate build (where we assume that there's no history)

但是将所有内容集中在一个阶段的替代方案并不吸引人

But the alternative of lumping everything together in one stage isn't appealing

stages:
  - test 

test:
  stage: test
  script:
    - if hash $HOME/miniconda/bin/conda 2>/dev/null; 
      then
         export PATH="$HOME/miniconda/bin:$PATH";
      else
        wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh;
        bash miniconda.sh -b -p $HOME/miniconda;
        export PATH="$HOME/miniconda/bin:$PATH";
      fi
    - conda update --yes conda
    - conda env create --quiet --force --file environment.yml
    - source activate myenv
    - nosetests --with-coverage --cover-erase --cover-package=mypackage --cover-html
    - pylint --reports=n tests/test_final.py
    - pep8 tests/test_final.py
    - grep pc_cov cover/index.html | egrep -o "[0-9]+\%" | awk '{ print "covered " $1;}'

我能想到的唯一其他选择是将环境创建步骤放在 before_script 阶段，但是在每个阶段之前不断地重新创建相同的环境似乎是多余的.

The only other option I can think of is to put the environment creation steps in a before_script stage, but it seems redundant to continuously recreate the same environment before each stage.

推荐答案

作业的独立性是设计特点.您可能已经注意到 GitLab 的界面允许您重新运行单个作业，如果作业相互依赖，这是不可能的.

The independence of the jobs is a design feature. You might have noticed that GitLab's interface allows you to re-run a single job which wouldn't be possible if the jobs depended on each other.

我不知道 Miniconda 究竟执行了什么，但如果它在特定文件夹中构建虚拟环境，您可以使用 cache 以在作业之间保留这些文件夹的内容.但是，您不能完全依赖它，因为文档指出...

I don't know what Miniconda exactly performs but if it builds a virtual environment in specific folders, you can use cache to preserve the content of those folders between the jobs. However, you cannot fully rely on it because the documentation states that...

缓存是尽最大努力提供的，所以不要期望缓存将始终存在.实现细节请查看GitLab Runner.

The cache is provided on a best-effort basis, so don't expect that the cache will be always present. For implementation details, please check GitLab Runner.

考虑到您的工作完全取决于正在构建的环境，您需要一种机制来检测(缓存的)环境是否存在并仅在需要时重新创建它.

Considering that your job absolutely depends on the environment being built, you would need a mechanism to detect whether the (cached) environment exists and re-create it only if needed.

我认为您正在尝试将环境设置和作业分开的好方法，因为如果您决定有一天运行不同的测试同时(同一阶段的作业并行运行).

I think you are taking good path trying to separate the environment setup and the jobs because it might save lots of time in case you decide one day to run different tests simultaneously (jobs at the same stage run in parallel).

这篇关于GitLab CI 在构建阶段之间保留环境的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！