如何从 Windows 主机上的 docker 容器内部执行 docker 命令 [英] how to execute docker command from inside docker container on windows host

问题描述

情况如下:我已经成功地在本地开发了一个超级简单的 ETL 过程，该过程从某个远程位置提取数据，然后将未处理的数据写入本地 Windows 机器上的 MongoDB 容器中.现在，我想使用 DockerOperator 为每个任务安排 Apache-Airflow 的这个过程，即我想为我的源代码创建一个 docker 镜像，然后使用 DockerOperator 在该镜像中执行源代码.由于我在 Windows 机器上工作，因此我只能使用 docker 容器中的 Airflow 来实际触发 Airflow DAG.Airflow 容器(以下称为 webserver)和 Mongo 容器(以下称为 mongo)都在 docker-compose.yml 文件中指定，您可以在最后看到.

The situation is the following: I have successfully, locally developed a super simple ETL process which extracts data from some remote location and then writes that unprocessed data into a MongoDB container on my local Windows machine. Now, I want to schedule this process with Apache-Airflow using the DockerOperator for every task, i.e. I want to create a docker image of my source code and then execute the source code in that image using the DockerOperator. Since I am working on a Windows machine, I can only use Airflow from within a docker container to actually trigger an Airflow DAG. Both the Airflow container (called webserver below) and Mongo container (called mongo below) are specified in a docker-compose.yml file, which you can see at the end.

据我所知，每次触发我的简单 ETL DAG 并执行 DockerOperator 时，网络服务器容器都会为每个 ETL 任务创建一个新的兄弟"容器，然后是这个新容器中的源代码执行并在任务完成后，再次删除这个新容器.如果到目前为止我的理解是正确的，则网络服务器容器需要能够执行 docker 命令，例如docker build... 能够创建这些兄弟容器.

To the best of my knowledge, every time my simple ETL DAG is triggered and the DockerOperator is executed, a new, "sibling" container for every ETL task is created by the webserver container, then the source code inside this new container is executed and after the task finished, this new container is deleted again. If my understanding is correct up to this point, the webserver container needs to be able to execute docker commands like e.g. docker build... to be able to create these sibling containers.

为了测试这个理论，我添加了卷 /var/run/docker.sock:/var/run/docker.sock 和 /usr/bin/docker:/usr/bin/docker 添加到 docker-compose.yml 文件中 webserver 容器的定义中，以便 webserver 容器可以在我的主机(windows)机器上使用 docker 守护进程.然后，我使用 docker-compose up -d 启动了 webserver 和 mongo 容器，我使用 docker exec -it 进入了 webserver 容器./bin/bash 然后我尝试了简单的 docker 命令 docker ps --all.然而，这个命令的输出是bash: docker: command not found.因此，似乎 Docker 没有正确安装在 webserver 容器中.如何确保 Docker 安装在 webserver 容器中，以便可以创建其他同级容器?

To test this theory, I added the volumes /var/run/docker.sock:/var/run/docker.sock and /usr/bin/docker:/usr/bin/docker to the definition of the webserver container in the docker-compose.yml file so that the webserver container can use the docker daemon on my host (windows) machine. Then, I started the webserver and mongo containers using docker-compose up -d, I entered into the webserver container using docker exec -it <name_of_webserver_container> /bin/bash and then I tried the simple docker command docker ps --all. However, the output of this command was bash: docker: command not found. So, it seems like Docker was not installed correctly inside the webserver container. How can I make sure Docker is installed inside the webserver container, so that other sibbling containers can be created?

您可以在下面找到用于 webserver 容器的 docker-compose.yml 文件和 Dockerfile 的相关方面.

Below you can find the relevant aspects of the docker-compose.yml file and the Dockerfile used for the webserver container.

docker-compose.yml 位于项目根目录:

webserver:
        build: ./docker-airflow
        restart: always
        privileged: true
        depends_on:
            - postgres  # some other service I cut out from this post
            - mongo
            - mongo-express  # some other service I cut out from this post
        environment:
            - LOAD_EX=n
            - EXECUTOR=Local
            - POSTGRES_USER=some_user
            - POSTGRES_PASSWORD=some_pw
            - POSTGRES_DB=airflowdb
        volumes:
            # DAG folder
            - ./docker-airflow/dags:/usr/local/airflow/dags
            # Add path for external python modules
            - ./src:/home/python_modules
            # Add path for airflow workspace folder
            - ./docker-airflow/workdir:/home/workdir
            # Mount the docker socket from the host (currently my laptop) into the webserver container
            - //var/run/docker.sock:/var/run/docker.sock  # double // are necessary for windows host
        ports:
            # Change port to 8081 to avoid Jupyter conflicts
            - 8081:8080
        command: webserver
        healthcheck:
            test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
            interval: 30s
            timeout: 30s
            retries: 3
        networks:
            - mynet

Dockerfile 用于位于 docker-airflow 文件夹中的网络服务器容器:

Dockerfile for the webserver container located in the docker-airflow folder:

FROM puckel/docker-airflow:1.10.4

# Adds DAG folder to the PATH
ENV PYTHONPATH "${PYTHONPATH}:/home/python_modules:/usr/local/airflow/dags"

# Install the optional packages and change the user to airflow again
COPY requirements.txt requirements.txt
USER root
RUN pip install -r requirements.txt

# Install docker inside the webserver container
RUN pip install -U pip && pip install docker
ENV SHARE_DIR /usr/local/share

# Install simple text editor for debugging
RUN ["apt-get", "update"]
RUN ["apt-get", "-y", "install", "vim"]

USER airflow

编辑/更新:

在合并了 Noe 的评论后，我将 webserver 容器的 Dockerfile 更改为以下内容:

After incorporating Noe's comments, I changed the Dockerfile of the webserver container to the following:

FROM puckel/docker-airflow:1.10.4

# Adds DAG folder to the PATH
ENV PYTHONPATH "${PYTHONPATH}:/home/python_modules:/usr/local/airflow/dags"

# Install the optional packages and change the user to airflow again
COPY requirements.txt requirements.txt
USER root
RUN pip install -r requirements.txt

# Install docker inside the webserver container
RUN curl -sSL https://get.docker.com/ | sh
ENV SHARE_DIR /usr/local/share

# Install simple text editor for debugging
RUN ["apt-get", "update"]
RUN ["apt-get", "-y", "install", "vim"]

USER airflow

并且我将 docker==4.1.0 添加到 requirements.txt 文件(在上面的 Dockerfile 中引用)，其中包含所有要安装的包网络服务器容器.

and I added docker==4.1.0 to the requirements.txt file (referenced in the above Dockerfile) which contains all to-be-installed packages inside the webserver container.

然而，当我第一次使用 docker-compose up --build -d 启动服务时，然后像这样进入 webserver 容器 docker exec -it ;/bin/bash 然后输入简单的docker命令docker ps --all，得到如下输出:

Now however, when I first start the services with docker-compose up --build -d, then enter into the webserver container like so docker exec -it <name_of_webserver_container> /bin/bash and then enter the simple docker command docker ps --all, I get the following output:

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json?all=1: dial unix /var/run/docker.sock: connect: permission denied

所以，似乎我仍然需要授予一些权利/特权，我觉得这很令人困惑，因为在 docker-compose.yml 文件的 webserver 部分，我已经把 特权:真.那么有人知道这个问题的原因吗?

So, seems like I still need to grant some rights/privileges, which I find confusing, because in the webserver section of the docker-compose.yml file, I have already put privileged: true. So does anyone know the cause of this problem?

编辑/更新/回答

从 webserver 容器的 Dockerfile 中删除 USER airlfow 后，我可以在 webserver 容器内执行 docker 命令！

After removing USER airlfow from the Dockerfile of the webserver container, I am able to docker commands inside the webserver container!

推荐答案

你正在尝试做的在 docker 中被称为 docker.

What you're trying to do is called docker in docker.

你需要做这些事情:

• 在容器中安装 docker 客户端

添加RUN curl -sSL https://get.docker.com/|sh

• 安装 docker 套接字

你挂载做得很好//var/run/docker.sock:/var/run/docker.sock

• 以特权模式运行您的容器

将 privileged: true 添加到您的容器

在您的具体情况下，您需要执行以下操作:

In your specific case you need to do these things:

• 删除 RUN pip install -U pip &&pip install docker 因为我们已经安装了
• 去掉USER气流，需要使用默认用户或者root用户
• 将docker==4.1.0添加到requirements.txt
• Remove RUN pip install -U pip && pip install docker because we already installed it
• Remove USER airflow, you need to use the default user or root user
• Add docker==4.1.0 to the requirements.txt

这篇关于如何从 Windows 主机上的 docker 容器内部执行 docker 命令的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！