GitLab CI在构建阶段之间保留环境 [英] GitLab CI preserve environment between build stages

问题描述

我正在研究一个python项目，并使用 miniconda 来管理我的环境.我正在使用具有以下运行程序配置的GitLab for CI

I am working on a python project and using miniconda to manage my environment. I am using GitLab for CI with the following runner configuration

stages:
  - build
  - test 

build:
  stage: build
  script:
    - if hash $HOME/miniconda/bin/conda 2>/dev/null; 
      then
         export PATH="$HOME/miniconda/bin:$PATH";
      else
        wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh;
        bash miniconda.sh -b -p $HOME/miniconda;
        export PATH="$HOME/miniconda/bin:$PATH";
      fi
    - conda update --yes conda

test:
  stage: test
  script:
    - conda env create --quiet --force --file environment.yml
    - source activate myenv
    - nosetests --with-coverage --cover-erase --cover-package=mypackage --cover-html
    - pylint --reports=n tests/test_final.py
    - pep8 tests/test_final.py
    - grep pc_cov cover/index.html | egrep -o "[0-9]+\%" | awk '{ print "covered " $1;}'

我(错误地)假设我的build阶段将设置可以运行我的test阶段的正确环境.查看这个问题和此GitLab问题我看到

I assumed (incorrectly) that my build stage would setup the correct environment in which I could run my test stage. Looking at this question and this GitLab issue I see that

.gitlab-ci.yml中定义的每个作业都作为单独的版本运行(我们在其中 假设没有历史)

each job defined in .gitlab-ci.yml is run as separate build (where we assume that there's no history)

但是将所有内容集中在一起的替代方法并不吸引人

But the alternative of lumping everything together in one stage isn't appealing

stages:
  - test 

test:
  stage: test
  script:
    - if hash $HOME/miniconda/bin/conda 2>/dev/null; 
      then
         export PATH="$HOME/miniconda/bin:$PATH";
      else
        wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh;
        bash miniconda.sh -b -p $HOME/miniconda;
        export PATH="$HOME/miniconda/bin:$PATH";
      fi
    - conda update --yes conda
    - conda env create --quiet --force --file environment.yml
    - source activate myenv
    - nosetests --with-coverage --cover-erase --cover-package=mypackage --cover-html
    - pylint --reports=n tests/test_final.py
    - pep8 tests/test_final.py
    - grep pc_cov cover/index.html | egrep -o "[0-9]+\%" | awk '{ print "covered " $1;}'

我唯一想到的另一种选择是将环境创建步骤放在

The only other option I can think of is to put the environment creation steps in a before_script stage, but it seems redundant to continuously recreate the same environment before each stage.

推荐答案

作业的独立性是一项设计功能.您可能已经注意到，GitLab的界面允许您重新运行单个作业，如果作业相互依赖，则将无法执行.

The independence of the jobs is a design feature. You might have noticed that GitLab's interface allows you to re-run a single job which wouldn't be possible if the jobs depended on each other.

我不知道Miniconda的确切功能，但是如果它在特定文件夹中构建虚拟环境，则可以使用

I don't know what Miniconda exactly performs but if it builds a virtual environment in specific folders, you can use cache to preserve the content of those folders between the jobs. However, you cannot fully rely on it because the documentation states that...

缓存是尽最大努力提供的，因此不要指望 缓存将始终存在.有关实施细节，请检查 亚搏体育app Runner.

The cache is provided on a best-effort basis, so don't expect that the cache will be always present. For implementation details, please check GitLab Runner.

考虑到您的工作完全取决于所构建的环境，您将需要一种机制来检测(缓存的)环境是否存在，并仅在需要时重新创建它.

Considering that your job absolutely depends on the environment being built, you would need a mechanism to detect whether the (cached) environment exists and re-create it only if needed.

我认为您正在尝试将环境设置和作业分开，因为如果您决定有一天同时运行不同的测试，这样可能会节省大量时间>(同一阶段的作业并行运行).

I think you are taking good path trying to separate the environment setup and the jobs because it might save lots of time in case you decide one day to run different tests simultaneously (jobs at the same stage run in parallel).

这篇关于GitLab CI在构建阶段之间保留环境的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！