在管道运行中使用环境类 [英] Using the Environment Class with Pipeline Runs

问题描述

我正在使用Environment类在管道中使用估算器步骤，以便拥有自定义Docker镜像，因为我需要一些 apt-get 包才能安装特定的pip包。与非管道版本的estimator相比，环境变量的docker部分从日志中似乎完全忽略了它。很简单，这似乎坏了：

I am using an estimator step for a pipeline using the Environment class, in order to have a custom Docker image as I need some apt-get packages to be able to install a specific pip package. It appears from the logs that it's completely ignoring, unlike the non-pipeline version of the estimator, the docker portion of the environment variable. Very simply, this seems broken :

我在SDK v1.0.65上运行，我的dockerfile被完全忽略了，我在使用

I'm running on SDK v1.0.65, and my dockerfile is completely ignored, I'm using

FROM mcr.microsoft.com/azureml/base:latest\nRUN apt-get update && apt-get -y install freetds-dev freetds-bin vim gcc

在我代码的base_dockerfile属性中。
这是我的代码的一部分：

in the base_dockerfile property of my code. Here's a snippet of my code :

from azureml.core import Environment
from azureml.core.environment import CondaDependencies
conda_dep = CondaDependencies()
conda_dep.add_pip_package('pymssql==2.1.1')
myenv = Environment(name="mssqlenv")
myenv.python.conda_dependencies=conda_dep
myenv.docker.enabled = True
myenv.docker.base_dockerfile = 'FROM mcr.microsoft.com/azureml/base:latest\nRUN apt-get update && apt-get -y install freetds-dev freetds-bin vim gcc'
myenv.docker.base_image = None

这在我自己使用估算器时效果很好，但是如果我在管道中插入此估算器，它将失败。这是从管道运行中启动它的代码：

This works well when I use an Estimator by itself, but if I insert this estimator in a Pipeline, it fails. Here's my code to launch it from a Pipeline run:

from azureml.pipeline.steps import EstimatorStep

sql_est_step = EstimatorStep(name="sql_step", 
                         estimator=est, 
                         estimator_entry_script_arguments=[],
                         runconfig_pipeline_params=None, 
                         compute_target=cpu_cluster)
from azureml.pipeline.core import Pipeline
from azureml.core import Experiment
pipeline = Pipeline(workspace=ws, steps=[sql_est_step])
pipeline_run = exp.submit(pipeline)

启动此命令时，容器构建服务的日志显示：

When launching this, the logs for the container building service reveal:

FROM continuumio/miniconda3:4.4.10... etc.

这表示它忽略了与该Estimator相关联的Environment类中的 FROM mcr .... 语句 pip install 失败。

Which indicates it's ignoring my FROM mcr.... statement in the Environment class I've associated with this Estimator, and my pip install fails.

我错过了什么吗？有解决方法吗？

Am I missing something? Is there a workaround?

推荐答案

我可以确认这是AML管道方面的错误。特别是，在管道作业中未正确传递runco​​nfig属性environment.docker.base_dockerfile。我们正在努力解决。同时，您可以使用以下线程中的变通方法：首先构建docker映像，并使用environment.docker.base_image（正确传递）进行指定。

I can confirm that this is a bug on the AML Pipeline side. Specifically, the runconfig property environment.docker.base_dockerfile is not being passed through correctly in pipeline jobs. We are working on a fix. In the meantime, you can use the workaround from this thread of building the docker image first and specifying it with environment.docker.base_image (which is passed through correctly).

这篇关于在管道运行中使用环境类的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！