无法在单节点kubernetes(在centos上)上通过"emptyDir"创建共享卷装载，但这在多节点k8s安装中有效 [英] cannot create a shared-volume mount via 'emptyDir' on single node kubernetes (on centos), but this works with multi-node k8s installation

问题描述

TL; DR-感谢Paul

如果遇到以下问题，最简单的解决方法是在运行Recipe引导单节点k8s之前执行以下命令:

If you have the problem described below, the easiest way to solve it is to execute the following command before running the Recipe to boot single node k8s:

   sudo chcon -Rt svirt_sandbox_file_t /var/lib/kubelet

原始问题描述

我正在尝试根据此食谱构建一个k8s环境> https://github.com/kubernetes /kubernetes/blob/release-1.1/docs/getting-started-guides/docker.md 为了进行集成测试，我们的代码库在k8s集群中配置了容器. 为了方便复制/粘贴，我将食谱中的所有命令都包含在下面的运行食谱"部分中

I am trying to put together a k8s environment based on this recipe > https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/getting-started-guides/docker.md for the purpose of integration testing our code base which provisions containers in a k8s cluster. For copy/paste convenience, I am including all the commands in the recipe in the section 'Run the Recipe', below

我有一个简单的复制控制器定义(在下面的复制控制器定义"中转载) 以获得非常标准的图像(nginx.)在此RC定义中，我尝试使用'emptyDir'挂载共享文件夹.
为简单起见，我在rep控制器定义中只有一个容器(因此，实际上并没有太多共享.)

I have a simple replication controller definition (reproduced below in 'Replication Controller Definition') for a very standard image (nginx.) In this RC definition I attempt to mount a shared folder using 'emptyDir'.
For simplicity I only have one container in the rep controller definition (so there really is not much sharing going on.)

现在，当我通过以下命令针对我们的多节点集群配置此RC时:'kubectl create -f shared.folder.json'
我能够登录到容器"nginx"并执行以下操作:

Now, when I provision this RC against our multi-node cluster via the command: 'kubectl create -f shared.folder.json'
I am able to log into the container 'nginx' and do the following:

touch /backup-folder/fooFile

我们的多节点集群的版本信息是:

The version info for our multi-node cluster is:

Server Version: 
    version.Info{
        Major:"1", 
        Minor:"1+", 
        GitVersion:"v1.1.3-beta.0.308+71b088a96ee101-dirty", 
        GitCommit:"71b088a96ee101967fc06e1f95b1cade8f6e30f9", GitTreeState:"dirty"}

但是...当我使用运行配方"中的步骤启动单节点k8s群集并进行配置时 使用命令'kubectl create -f shared.folder.json'针对该集群，然后生成一个bash shell 在nginx容器中并尝试与上述相同的touch命令，但是在单节点情况下，我得到一个 错误:触摸:无法触摸'/backup-folder/fooo':权限被拒绝

HOWEVER... when I bring up a single node k8s cluster using the steps in 'Run the Recipe', and provision against that cluster using the command 'kubectl create -f shared.folder.json' I then spawn a bash shell in the nginx container and attempt the same touch command as above, but in the single node case I get an error: touch: cannot touch '/backup-folder/fooo': Permission denied

在两种情况下，我从运行mount -l得到的信息是有用的:

In case it is useful here is the info I get from running mount -l in the two cases:

1)单节点k8s

  root@foo-hzxd6:/# mount -l  | grep backup-folder
  /dev/mapper/cl-root on /backup-folder type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

2)多节点k8s

  root@foo-vcbc9:/# mount -l  | grep backup-folder
 /dev/vdb on /backup-folder type ext3 (rw,relatime,data=ordered)

复制控制器定义

Replication Controller Definition

shared.folder.json

{
  "kind": "ReplicationController",
  "apiVersion": "v1",
  "metadata":{
      "name":"foo",
      "labels":{
         "app":"foo",
         "role":"foo"
      }
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "name": "nginx"
    },
    "template": {
          "metadata": {
            "name": "nginx",
            "labels": {
              "name": "nginx"
            }
          },
          "spec": {
            "containers": [
              {
                "name": "nginx",
                "image": "nginx",
                        "imagePullPolicy": "Always",
                "ports": [
                  {
                    "containerPort": 8080
                  },
                  {
                    "containerPort": 8081
                  }
                ],
                        "command": ["sleep", "10000"],
                        "volumeMounts": [
                      {
                        "name": "shared-volume",
                        "mountPath": "/backup-folder"
                      }
                    ]
              }
            ],
            "volumes": [
              {
                "name": "shared-volume",
                "emptyDir": { }
              }
            ]
          }
    }
  }
} 

运行配方

Run the Recipe

docker run --net=host -d gcr.io/google_containers/etcd:2.0.12 /usr/local/bin/etcd --addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001 --data-dir=/var/etcd/data


docker run \
    --volume=/:/rootfs:ro \
    --volume=/sys:/sys:ro \
    --volume=/dev:/dev \
    --volume=/var/lib/docker/:/var/lib/docker:ro \
    --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
    --volume=/var/run:/var/run:rw \
    --net=host \
    --pid=host \
    --privileged=true \
    -d \
    gcr.io/google_containers/hyperkube:v1.0.1 \
    /hyperkube kubelet --containerized --hostname-override="127.0.0.1" --address="0.0.0.0" --api-servers=http://localhost:8080 --config=/etc/kubernetes/manifests


docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube proxy --master=http://127.0.0.1:8080 --v=2

旁注

(我们的k8s版本-我们用于多节点情况-在'stock'kubernetes的基础上进行了一些更改，但是我敢肯定这些更改都与挂载文件夹无关).

( our version of k8s --which we use for the multi node case -- has some changes layered on top of 'stock' kubernetes, but i'm pretty sure none of these changes has to do with mounting folders).

结束语-有关SELinux拒绝访问共享文件夹的更多详细信息

为响应Paul的详细要求，我们开始:

In answer to Paul's request for more detail, here we go:

首先，通过"setenforce 1"将执行权转回是"

First, flip enforcement back to 'yes' via: "setenforce 1"

接下来，杀死所有docker容器，然后通过上面提供的3个步骤配方重新启动k8s单节点.

Next, kill all docker containers, then relaunch k8s single node via 3 step recipe provided above.

接下来，通过"kubectl create -f shared.folder.json"配置Pod.

Next, provision pod via "kubectl create -f shared.folder.json"

接下来，通过以下方式将外壳扔到容器中:"kubectl exec -i -t foo-podxxx -c nginx-bash"

Next, throw a shell into container via: "kubectl exec -i -t foo-podxxx -c nginx -- bash "

在bash shell中:触摸/backup-folder/blah"

In bash shell: "touch /backup-folder/blah"

结果:

> sudo ausearch -ts recent -m AVC
----
time->Tue Jan 19 11:33:19 2016
type=SYSCALL msg=audit(1453231999.925:865015): arch=c000003e syscall=2 success=no exit=-13 a0=7ffd65fc1e45 a1=941 a2=1b6 a3=7ffd65fc09f0 items=0 ppid=25089 pid=25127 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts4 ses=4294967295 comm="touch" exe="/bin/touch" subj=system_u:system_r:svirt_lxc_net_t:s0:c202,c694 key=(null)
type=AVC msg=audit(1453231999.925:865015): avc:  denied  { create } for  pid=25127 comm="touch" name="blah" scontext=system_u:system_r:svirt_lxc_net_t:s0:c202,c694 tcontext=system_u:object_r:docker_var_lib_t:s0 tclass=file
(backup-agent-scripts) /home/chris/dev/krylov/scripts > 

kubelet的日志: https://dl.dropboxusercontent.com/u/9940067/kubelet.log

Log of kubelet: https://dl.dropboxusercontent.com/u/9940067/kubelet.log

推荐答案

以下是完整的单节点k8s启动脚本，该脚本使我的问题消除了. 感谢Paul Morie为我提供了解决方案(脚本中神奇的第一行).

Here is the complete single node k8s start up script that makes my problem go away. Thanks to Paul Morie for providing me w/ the solution (the magic first line in the script).

更新 这是Paul向我发送的有关为什么使用chcon的更新: 基本上，它所做的是更改包含所有Pod的卷目录的SELinux类型 卷到svirt_sandbox_file_t，这是大多数SELinux策略允许容器使用的上下文 (通常与svirt_lxc_net_t一起运行)使用.
因此，TLDR，该命令使docker容器可以使用kube卷目录(当然，容器 只能访问在其容器中消耗并安装到容器中的卷.

Update Here is an update that Paul sent me on why chcon is used: basically what it does is change the SELinux type for the volume directory that holds all the pod volumes to svirt_sandbox_file_t, which is the context that most SELinux policies allow containers (typically running with svirt_lxc_net_t) to use.
So, TLDR, that command makes the kube volume directory usable by docker containers (though of course containers only have access to the volumes that are consumed in their pod and then mounted into the container).

我对此的理解是，通常Docker容器是独立运行的，彼此之间看不见 文件系统中，chcon允许我们以受控方式打破这种隔离，从而 仅使用卷安装指令才允许进行这种共享. 此说明似乎很重要.

My understanding of this is that normally Docker container run in isolation and can't see each others file systems, the chcon allows us to break this isolation, in a controlled fashion, such that only using volume mount directives is this sharing allowed to happen. This explanation seems relevant.

#   magic selinux context set command is required. for details, see: http://stackoverflow.com/questions/34777111/cannot-create-a-shared-volume-mount-via-emptydir-on-single-node-kubernetes-on
#
sudo chcon -Rt svirt_sandbox_file_t /var/lib/kubelet


docker run --net=host -d gcr.io/google_containers/etcd:2.0.12 /usr/local/bin/etcd --addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001 --data-dir=/var/etcd/data


docker run \
    --volume=/:/rootfs:ro \
    --volume=/sys:/sys:ro \
    --volume=/dev:/dev \
    --volume=/var/lib/docker/:/var/lib/docker:ro \
    --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
    --volume=/var/run:/var/run:rw \
    --net=host \
    --pid=host \
    --privileged=true \
    -d \
    gcr.io/google_containers/hyperkube:v1.0.1 \
    /hyperkube kubelet --containerized --hostname-override="127.0.0.1" --address="0.0.0.0" --api-servers=http://localhost:8080 --config=/etc/kubernetes/manifests

docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube proxy --master=http://127.0.0.1:8080 --v=2

sleep 20   # give everything time to launch

这篇关于无法在单节点kubernetes(在centos上)上通过"emptyDir"创建共享卷装载，但这在多节点k8s安装中有效的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！