如何在多个Pod上安装相同的持久卷? [英] How can I mount the same persistent volume on multiple pods?

问题描述

我有一个三节点GCE集群和一个带有三个副本的单容器GKE部署.我这样创建了PV和PVC:

# Create a persistent volume for web content
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nginx-content
  labels:
    type: local
spec:
  capacity:
    storage: 5Gi
  accessModes:
   - ReadOnlyMany
  hostPath:
    path: "/usr/share/nginx/html"
--
# Request a persistent volume for web content
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-content-claim
  annotations:
    volume.alpha.kubernetes.io/storage-class: default
spec:
  accessModes: [ReadOnlyMany]
  resources:
    requests:
      storage: 5Gi

在容器规范中引用了它们，如下所示:

    spec:
      containers:
      - image: launcher.gcr.io/google/nginx1
        name: nginx-container
        volumeMounts:
          - name: nginx-content
            mountPath: /usr/share/nginx/html
        ports:
          - containerPort: 80
      volumes:
      - name: nginx-content
        persistentVolumeClaim:
          claimName: nginx-content-claim

即使我将卷创建为ReadOnlyMany，在任何给定时间也只能安装一个Pod.其余的给出错误400:RESOURCE_IN_USE_BY_ANOTHER_RESOURCE".我该如何做，以便所有三个副本从同一卷中读取相同的Web内容?

解决方案

首先，我想指出您的配置中的一个基本差异.请注意，当您使用示例中定义的PersistentVolumeClaim时，根本就不会使用nginx-content PersistentVolume.您可以通过运行以下命令轻松地进行验证:

kubectl get pv

在您的 GKE集群上.您会注意到，除了手动创建的nginx-content PV外，还有另一种是根据您应用的PVC自动置备的.

请注意，在您的PersistentVolumeClaim定义中，您明确地引用了default存储类，该类与您手动创建的PV无关.实际上，即使您完全省略了注释，也可以:

annotations:
        volume.alpha.kubernetes.io/storage-class: default

它的工作方式完全相同，即无论如何都将使用default存储类.在 GKE 上使用默认存储类意味着 GCE永久磁盘将用作您的卷配置程序.您可以在此处了解更多信息:

已配置诸如gcePersistentDisk之类的卷实现 通过StorageClass资源. GKE为创建一个默认的StorageClass 您使用标准持久性磁盘类型(ext4).默认值 当PersistentVolumeClaim未指定存储空间时使用StorageClass StorageClassName.您可以替换提供的默认StorageClass 与你自己的.

但是让我们继续解决您面临的问题.

解决方案:

首先，我想强调您不必使用任何类似NFS的文件系统即可达到目标.

如果需要在ReadOnlyMany模式下可以使用PersistentVolume，则 GCE永久磁盘是完全满足您要求的完美解决方案.

许多Pods可以同时将其安装在ro模式下，许多Pods更重要的是，它们安排在不同的 GKE nodes上.此外，它的配置非常简单，并且可以直接在 GKE 上运行.

如果您想以ReadWriteMany模式使用存储，我同意NFS之类的东西可能是唯一的解决方案，因为 GCE永久磁盘不提供这种功能.

让我们仔细看看我们如何配置它.

我们需要从定义PVC开始.此步骤实际上已经由您自己完成，但是您在后续步骤中迷失了一些.让我解释一下它是如何工作的.

以下配置正确(正如我提到的annotations部分可以省略):

# Request a persistent volume for web content
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-content-claim
spec:
  accessModes: [ReadOnlyMany]
  resources:
    requests:
      storage: 5Gi

但是，我想对此添加一条重要的评论.你说:

即使我将卷创建为ReadOnlyMany，也只有一个Pod可以 在任何给定时间安装该卷.

好吧，实际上您没有.我知道这似乎有些棘手，有些令人惊讶，但这并不是定义accessModes真正起作用的方式.实际上，这是一个被广泛误解的概念.首先，您无法在PVC 中定义访问模式，这意味着要放置所需的约束.支持的访问模式是特定存储类型的固有功能.它们已经由存储提供商定义.

您在PVC定义中实际执行的操作是请求支持特定访问模式的PV.请注意，它采用列表的形式，这意味着您可以提供许多您希望PV支持的访问模式.

基本上，这就像说:"嘿！存储提供商！给我一个支持ReadOnlyMany模式的卷." 您正在用这种方式要求一种满足您要求的存储.但是请记住，您可以得到比要求更多的东西.这也是我们在 GCP 中要求支持ReadOnlyMany模式的PV时的情况.它为我们创建了一个PersistentVolume，它满足我们在accessModes部分中列出的要求，但它也支持ReadWriteOnce模式.尽管我们并没有要求也支持ReadWriteOnce的东西，但您可能会同意我的观点，即内置支持这两种模式的存储完全可以满足我们对支持ReadOnlyMany的要求.所以基本上这就是它的工作方式.

由GCP自动为您的PVC响应提供的PV支持这两个accessModes，并且如果您未在Pod或Deployment定义中明确指定要挂载的accessModes 只读模式，默认情况下，它以读写模式安装.

您可以通过将其附加到能够成功安装PersistentVolume的Pod来轻松地对其进行验证:

kubectl exec -ti pod-name -- /bin/bash

并尝试在已挂载的文件系统上写一些东西.

您收到的错误消息:

"Error 400: RESOURCE_IN_USE_BY_ANOTHER_RESOURCE"

特别关注 GCE永久磁盘，该磁盘已经由一个 GKE node在ReadWriteOnce模式下安装，并且不能由另一个node装载，该node Pods的其余部分已排定.

如果希望以ReadOnlyMany模式安装它，则需要在Deployment定义中明确指定它，方法是在Pod's模板规范下的volumes部分中添加readOnly: true语句，如下所示:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: nginx-content
      volumes:
      - name: nginx-content
        persistentVolumeClaim:
          claimName: nginx-content-claim
          readOnly: true

但是请记住，要能够以readOnly模式安装它，首先我们需要使用数据预先填充该卷.否则，您将看到另一条错误消息，提示未格式化的卷不能以只读模式挂载.

最简单的方法是创建一个Pod，该Pod仅用于将已经上传到我们的 GKE节点之一的数据复制到我们的目的地PV.

请注意，可以用许多不同的方法来预填充PersistentVolume数据.您只能在将要在Deployment中使用的PersistentVolume装载在Pod中，并使用curl或wget从某个外部位置获取数据，并将其直接保存在目标PV上.由你决定.

在我的示例中，我演示了如何使用其他本地 a>卷，该卷允许我们将directory，partition或disk装入我们的一个kubernetes上可用的directory，partition或disk(在我的示例中，我使用一个位于我的GKE节点上的目录/var/tmp/test)节点.它比hostPath更加灵活，因为我们不必担心将Pod调度到包含数据的特定节点. PersistentVolume中已经定义了特定的节点关联性规则，并且Pod是自动在特定节点上安排的.

要创建它，我们需要三件事:

StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

PersistentVolume定义:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-pv
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /var/tmp/test
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - <gke-node-name>

，最后是PersistentVolumeClaim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Gi
  storageClassName: local-storage

然后，我们可以创建临时Pod，该临时Pod仅用于将数据从 GKE节点复制到我们的 GCE永久磁盘.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
    - name: myfrontend
      image: nginx
      volumeMounts:
      - mountPath: "/mnt/source"
        name: mypd
      - mountPath: "/mnt/destination"
        name: nginx-content
  volumes:
    - name: mypd
      persistentVolumeClaim:
        claimName: myclaim
    - name: nginx-content
      persistentVolumeClaim:
        claimName: nginx-content-claim

您在上面看到的路径并不是很重要.此Pod的任务只是允许我们将数据复制到目标PV.最终，我们的PV将安装在完全不同的路径上.

一旦创建了Pod并且成功安装了两个卷，我们可以通过运行以下命令来附加它:

kubectl exec -ti my-pod -- /bin/bash

只需运行Pod，即可运行:

cp /mnt/source/* /mnt/destination/

仅此而已.现在我们可以exit并删除我们的临时Pod:

kubectl delete pod mypod

一旦它消失了，我们可以应用我们的Deployment，并且我们的PersistentVolume最终可以通过位于 GKE节点上的所有Pods以readOnly模式安装:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: nginx-content
      volumes:
      - name: nginx-content
        persistentVolumeClaim:
          claimName: nginx-content-claim
          readOnly: true

顺便说一句.如果您可以确定Pods仅在一个特定节点上进行调度的事实，则可以完全放弃使用 GCE永久磁盘并切换到上述

They are referenced in the container spec like so:

    spec:
      containers:
      - image: launcher.gcr.io/google/nginx1
        name: nginx-container
        volumeMounts:
          - name: nginx-content
            mountPath: /usr/share/nginx/html
        ports:
          - containerPort: 80
      volumes:
      - name: nginx-content
        persistentVolumeClaim:
          claimName: nginx-content-claim

Even though I created the volumes as ReadOnlyMany, only one pod can mount the volume at any given time. The rest give "Error 400: RESOURCE_IN_USE_BY_ANOTHER_RESOURCE". How can I make it so all three replicas read the same web content from the same volume?

解决方案

First I'd like to point out one fundamental discrapency in your configuration. Note that when you use your PersistentVolumeClaim defined as in your example, you don't use your nginx-content PersistentVolume at all. You can easily verify it by running:

kubectl get pv

on your GKE cluster. You'll notice that apart from your manually created nginx-content PV, there is another one, which was automatically provisioned based on the PVC that you applied.

Note that in your PersistentVolumeClaim definition you're explicitely referring the default storage class which has nothing to do with your manually created PV. Actually even if you completely omit the annotation:

annotations:
        volume.alpha.kubernetes.io/storage-class: default

it will work exactly the same way, namely the default storage class will be used anyway. Using the default storage class on GKE means that GCE Persistent Disk will be used as your volume provisioner. You can read more about it here:

Volume implementations such as gcePersistentDisk are configured through StorageClass resources. GKE creates a default StorageClass for you which uses the standard persistent disk type (ext4). The default StorageClass is used when a PersistentVolumeClaim doesn't specify a StorageClassName. You can replace the provided default StorageClass with your own.

But let's move on to the solution of the problem you're facing.

Solution:

First, I'd like to emphasize you don't have to use any NFS-like filesystems to achive your goal.

If you need your PersistentVolume to be available in ReadOnlyMany mode, GCE Persistent Disk is a perfect solution that entirely meets your requirements.

It can be mounted in ro mode by many Pods at the same time and what is even more important by many Pods, scheduled on different GKE nodes. Furthermore it's really simple to configure and it works on GKE out of the box.

In case you want to use your storage in ReadWriteMany mode, I agree that something like NFS may be the only solution as GCE Persistent Disk doesn't provide such capability.

Let's take a closer look how we can configure it.

We need to start from defining our PVC. This step was actually already done by yourself but you got lost a bit in further steps. Let me explain how it works.

The following configuration is correct (as I mentioned annotations section can be omitted):

# Request a persistent volume for web content
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-content-claim
spec:
  accessModes: [ReadOnlyMany]
  resources:
    requests:
      storage: 5Gi

However I'd like to add one important comment to this. You said:

Even though I created the volumes as ReadOnlyMany, only one pod can mount the volume at any given time.

Well, actually you didn't. I know it may seem a bit tricky and somewhat surprising but this is not the way how defining accessModes really works. In fact it's a widely misunderstood concept. First of all you cannot define access modes in PVC in a sense of putting there the constraints you want. Supported access modes are inherent feature of a particular storage type. They are already defined by the storage provider.

What you actually do in PVC definition is requesting a PV that supports the particular access mode or access modes. Note that it's in a form of a list which means you may provide many different access modes that you want your PV to support.

Basically it's like saying: "Hey! Storage provider! Give me a volume that supports ReadOnlyMany mode." You're asking this way for a storage that will satisfy your requirements. Keep in mind however that you can be given more than you ask. And this is also our scenario when asking for a PV that supports ReadOnlyMany mode in GCP. It creates for us a PersistentVolume which meets our requirements we listed in accessModes section but it also supports ReadWriteOnce mode. Although we didn't ask for something that also supports ReadWriteOnce you will probably agree with me that storage which has a built-in support for those two modes fully satisfies our request for something that supports ReadOnlyMany. So basically this is the way it works.

Your PV that was automatically provisioned by GCP in response for your PVC supports those two accessModes and if you don't specify explicitely in Pod or Deployment definition that you want to mount it in read-only mode, by default it is mounted in read-write mode.

You can easily verify it by attaching to the Pod that was able to successfully mount the PersistentVolume:

kubectl exec -ti pod-name -- /bin/bash

and trying to write something on the mounted filesystem.

The error message you get:

"Error 400: RESOURCE_IN_USE_BY_ANOTHER_RESOURCE"

concerns specifically GCE Persistent Disk that is already mounted by one GKE node in ReadWriteOnce mode and it cannot be mounted by another node on which the rest of your Pods were scheduled.

If you want it to be mounted in ReadOnlyMany mode, you need to specify it explicitely in your Deployment definition by adding readOnly: true statement in the volumes section under Pod's template specification like below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: nginx-content
      volumes:
      - name: nginx-content
        persistentVolumeClaim:
          claimName: nginx-content-claim
          readOnly: true

Keep in mind however that to be able to mount it in readOnly mode, first we need to pre-populate such volume with data. Otherwise you'll see another error message, saying that unformatted volume cannot be mounted in read only mode.

The easiest way to do it is by creating a single Pod which will serve only for copying data which was already uploaded to one of our GKE nodes to our destination PV.

Note that pre-populating PersistentVolume with data can be done in many different ways. You can mount in such Pod only your PersistentVolume that you will be using in your Deployment and get your data using curl or wget from some external location saving it directly on your destination PV. It's up to you.

In my example I'm showing how to do it using additional local volume that allows us to mount into our Pod a directory, partition or disk (in my example I use a directory /var/tmp/test located on one of my GKE nodes) available on one of our kubernetes nodes. It's much more flexible solution than hostPath as we don't have to care about scheduling such Pod to particular node, that contains the data. Specific node affinity rule is already defined in PersistentVolume and Pod is automatically scheduled on specific node.

To create it we need 3 things:

StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

PersistentVolume definition:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-pv
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /var/tmp/test
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - <gke-node-name>

and finally PersistentVolumeClaim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Gi
  storageClassName: local-storage

Then we can create our temporary Pod which will serve only for copying data from our GKE node to our GCE Persistent Disk.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
    - name: myfrontend
      image: nginx
      volumeMounts:
      - mountPath: "/mnt/source"
        name: mypd
      - mountPath: "/mnt/destination"
        name: nginx-content
  volumes:
    - name: mypd
      persistentVolumeClaim:
        claimName: myclaim
    - name: nginx-content
      persistentVolumeClaim:
        claimName: nginx-content-claim

Paths you can see above are not really important. The task of this Pod is only to allow us to copy our data to the destination PV. Eventually our PV will be mounted in completely different path.

Once the Pod is created and both volumes are successfully mounted, we can attach to it by running:

kubectl exec -ti my-pod -- /bin/bash

Withing the Pod simply run:

cp /mnt/source/* /mnt/destination/

That's all. Now we can exit and delete our temporary Pod:

kubectl delete pod mypod

Once it is gone, we can apply our Deployment and our PersistentVolume finally can be mounted in readOnly mode by all the Pods located on various GKE nodes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: nginx-content
      volumes:
      - name: nginx-content
        persistentVolumeClaim:
          claimName: nginx-content-claim
          readOnly: true

Btw. if you are ok with the fact that your Pods will be scheduled only on one particular node, you can give up on using GCE Persistent Disk at all and switch to the above mentioned local volume. This way all your Pods will be able not only to read from it but also to write to it at the same time. The only caveat is that all those Pods will be running on a single node.

这篇关于如何在多个Pod上安装相同的持久卷?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！