如何在多个Pod上安装相同的持久卷? [英] How can I mount the same persistent volume on multiple pods?
问题描述
我有一个三节点GCE集群和一个带有三个副本的单容器GKE部署.我这样创建了PV和PVC:
# Create a persistent volume for web content
apiVersion: v1
kind: PersistentVolume
metadata:
name: nginx-content
labels:
type: local
spec:
capacity:
storage: 5Gi
accessModes:
- ReadOnlyMany
hostPath:
path: "/usr/share/nginx/html"
--
# Request a persistent volume for web content
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nginx-content-claim
annotations:
volume.alpha.kubernetes.io/storage-class: default
spec:
accessModes: [ReadOnlyMany]
resources:
requests:
storage: 5Gi
在容器规范中引用了它们,如下所示:
spec:
containers:
- image: launcher.gcr.io/google/nginx1
name: nginx-container
volumeMounts:
- name: nginx-content
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
volumes:
- name: nginx-content
persistentVolumeClaim:
claimName: nginx-content-claim
即使我将卷创建为ReadOnlyMany,在任何给定时间也只能安装一个Pod.其余的给出错误400:RESOURCE_IN_USE_BY_ANOTHER_RESOURCE".我该如何做,以便所有三个副本从同一卷中读取相同的Web内容?
首先,我想指出您的配置中的一个基本差异.请注意,当您使用示例中定义的PersistentVolumeClaim
时,根本就不会使用nginx-content
PersistentVolume
.您可以通过运行以下命令轻松地进行验证:
kubectl get pv
在您的 GKE集群上.您会注意到,除了手动创建的nginx-content
PV
外,还有另一种是根据您应用的PVC
自动置备的.
请注意,在您的PersistentVolumeClaim
定义中,您明确地引用了default
存储类,该类与您手动创建的PV
无关.实际上,即使您完全省略了注释,也可以:
annotations:
volume.alpha.kubernetes.io/storage-class: default
它的工作方式完全相同,即无论如何都将使用default
存储类.在 GKE 上使用默认存储类意味着 GCE永久磁盘将用作您的卷配置程序.您可以在此处了解更多信息:>
已配置诸如gcePersistentDisk之类的卷实现 通过StorageClass资源. GKE为创建一个默认的StorageClass 您使用标准持久性磁盘类型(ext4).默认值 当PersistentVolumeClaim未指定存储空间时使用StorageClass StorageClassName.您可以替换提供的默认StorageClass 与你自己的.
但是让我们继续解决您面临的问题.
解决方案:
首先,我想强调您不必使用任何类似NFS的文件系统即可达到目标.
如果需要在ReadOnlyMany
模式下可以使用PersistentVolume
,则 GCE永久磁盘是完全满足您要求的完美解决方案.
许多Pods
可以同时将其安装在ro
模式下,许多Pods
更重要的是,它们安排在不同的 GKE nodes
上.此外,它的配置非常简单,并且可以直接在 GKE 上运行.
如果您想以ReadWriteMany
模式使用存储,我同意NFS之类的东西可能是唯一的解决方案,因为 GCE永久磁盘不提供这种功能.
让我们仔细看看我们如何配置它.
我们需要从定义PVC
开始.此步骤实际上已经由您自己完成,但是您在后续步骤中迷失了一些.让我解释一下它是如何工作的.
以下配置正确(正如我提到的annotations
部分可以省略):
# Request a persistent volume for web content
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nginx-content-claim
spec:
accessModes: [ReadOnlyMany]
resources:
requests:
storage: 5Gi
但是,我想对此添加一条重要的评论.你说:
即使我将卷创建为ReadOnlyMany,也只有一个Pod可以 在任何给定时间安装该卷.
好吧,实际上您没有.我知道这似乎有些棘手,有些令人惊讶,但这并不是定义accessModes
真正起作用的方式.实际上,这是一个被广泛误解的概念.首先,您无法在PVC
中定义访问模式,这意味着要放置所需的约束.支持的访问模式是特定存储类型的固有功能.它们已经由存储提供商定义.
您在PVC
定义中实际执行的操作是请求支持特定访问模式的PV
.请注意,它采用列表的形式,这意味着您可以提供许多您希望PV
支持的访问模式.
基本上,这就像说:"嘿!存储提供商!给我一个支持ReadOnlyMany
模式的卷." 您正在用这种方式要求一种满足您要求的存储.但是请记住,您可以得到比要求更多的东西.这也是我们在 GCP 中要求支持ReadOnlyMany
模式的PV
时的情况.它为我们创建了一个PersistentVolume
,它满足我们在accessModes
部分中列出的要求,但它也支持ReadWriteOnce
模式.尽管我们并没有要求也支持ReadWriteOnce
的东西,但您可能会同意我的观点,即内置支持这两种模式的存储完全可以满足我们对支持ReadOnlyMany
的要求.所以基本上这就是它的工作方式.
由GCP自动为您的PVC
响应提供的PV
支持这两个accessModes
,并且如果您未在Pod
或Deployment
定义中明确指定要挂载的accessModes
只读模式,默认情况下,它以读写模式安装.
您可以通过将其附加到能够成功安装PersistentVolume
的Pod
来轻松地对其进行验证:
kubectl exec -ti pod-name -- /bin/bash
并尝试在已挂载的文件系统上写一些东西.
您收到的错误消息:
"Error 400: RESOURCE_IN_USE_BY_ANOTHER_RESOURCE"
特别关注 GCE永久磁盘,该磁盘已经由一个 GKE node
在ReadWriteOnce
模式下安装,并且不能由另一个node
装载,该node
Pods
的其余部分已排定.
如果希望以ReadOnlyMany
模式安装它,则需要在Deployment
定义中明确指定它,方法是在Pod's
模板规范下的volumes
部分中添加readOnly: true
语句,如下所示:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: nginx-content
volumes:
- name: nginx-content
persistentVolumeClaim:
claimName: nginx-content-claim
readOnly: true
但是请记住,要能够以readOnly
模式安装它,首先我们需要使用数据预先填充该卷.否则,您将看到另一条错误消息,提示未格式化的卷不能以只读模式挂载.
最简单的方法是创建一个Pod
,该Pod
仅用于将已经上传到我们的 GKE节点之一的数据复制到我们的目的地PV
.>
请注意,可以用许多不同的方法来预填充PersistentVolume
数据.您只能在将要在Deployment
中使用的PersistentVolume
装载在Pod
中,并使用curl
或wget
从某个外部位置获取数据,并将其直接保存在目标PV
上.由你决定.
要创建它,我们需要三件事:
StorageClass
:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
PersistentVolume
定义:
apiVersion: v1
kind: PersistentVolume
metadata:
name: example-pv
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /var/tmp/test
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- <gke-node-name>
,最后是PersistentVolumeClaim
:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myclaim
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 10Gi
storageClassName: local-storage
然后,我们可以创建临时Pod
,该临时Pod
仅用于将数据从 GKE节点复制到我们的 GCE永久磁盘.
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: myfrontend
image: nginx
volumeMounts:
- mountPath: "/mnt/source"
name: mypd
- mountPath: "/mnt/destination"
name: nginx-content
volumes:
- name: mypd
persistentVolumeClaim:
claimName: myclaim
- name: nginx-content
persistentVolumeClaim:
claimName: nginx-content-claim
您在上面看到的路径并不是很重要.此Pod
的任务只是允许我们将数据复制到目标PV
.最终,我们的PV
将安装在完全不同的路径上.
一旦创建了Pod
并且成功安装了两个卷,我们可以通过运行以下命令来附加它:
kubectl exec -ti my-pod -- /bin/bash
只需运行Pod
,即可运行:
cp /mnt/source/* /mnt/destination/
仅此而已.现在我们可以exit
并删除我们的临时Pod
:
kubectl delete pod mypod
一旦它消失了,我们可以应用我们的Deployment
,并且我们的PersistentVolume
最终可以通过位于 GKE节点上的所有Pods
以readOnly
模式安装:>
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: nginx-content
volumes:
- name: nginx-content
persistentVolumeClaim:
claimName: nginx-content-claim
readOnly: true
顺便说一句.如果您可以确定 They are referenced in the container spec like so: Even though I created the volumes as ReadOnlyMany, only one pod can mount the volume at any given time. The rest give "Error 400: RESOURCE_IN_USE_BY_ANOTHER_RESOURCE". How can I make it so all three replicas read the same web content from the same volume? First I'd like to point out one fundamental discrapency in your configuration. Note that when you use your on your GKE cluster. You'll notice that apart from your manually created Note that in your it will work exactly the same way, namely the Volume implementations such as gcePersistentDisk are configured
through StorageClass resources. GKE creates a default StorageClass for
you which uses the standard persistent disk type (ext4). The default
StorageClass is used when a PersistentVolumeClaim doesn't specify a
StorageClassName. You can replace the provided default StorageClass
with your own. But let's move on to the solution of the problem you're facing. First, I'd like to emphasize you don't have to use any NFS-like filesystems to achive your goal. If you need your It can be mounted in In case you want to use your storage in Let's take a closer look how we can configure it. We need to start from defining our The following configuration is correct (as I mentioned However I'd like to add one important comment to this. You said: Even though I created the volumes as ReadOnlyMany, only one pod can
mount the volume at any given time. Well, actually you didn't. I know it may seem a bit tricky and somewhat surprising but this is not the way how defining What you actually do in Basically it's like saying: "Hey! Storage provider! Give me a volume that supports Your You can easily verify it by attaching to the and trying to write something on the mounted filesystem. The error message you get: concerns specifically GCE Persistent Disk that is already mounted by one GKE If you want it to be mounted in Keep in mind however that to be able to mount it in The easiest way to do it is by creating a single Note that pre-populating In my example I'm showing how to do it using additional local volume that allows us to mount into our To create it we need 3 things: and finally Then we can create our temporary Paths you can see above are not really important. The task of this Once the Withing the That's all. Now we can Once it is gone, we can apply our Btw. if you are ok with the fact that your 这篇关于如何在多个Pod上安装相同的持久卷?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!Pods
仅在一个特定节点上进行调度的事实,则可以完全放弃使用 GCE永久磁盘并切换到上述 spec:
containers:
- image: launcher.gcr.io/google/nginx1
name: nginx-container
volumeMounts:
- name: nginx-content
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
volumes:
- name: nginx-content
persistentVolumeClaim:
claimName: nginx-content-claim
PersistentVolumeClaim
defined as in your example, you don't use your nginx-content
PersistentVolume
at all. You can easily verify it by running:kubectl get pv
nginx-content
PV
, there is another one, which was automatically provisioned based on the PVC
that you applied.PersistentVolumeClaim
definition you're explicitely referring the default
storage class which has nothing to do with your manually created PV
. Actually even if you completely omit the annotation:annotations:
volume.alpha.kubernetes.io/storage-class: default
default
storage class will be used anyway. Using the default storage class on GKE means that GCE Persistent Disk will be used as your volume provisioner. You can read more about it here:
Solution:
PersistentVolume
to be available in ReadOnlyMany
mode, GCE Persistent Disk is a perfect solution that entirely meets your requirements.ro
mode by many Pods
at the same time and what is even more important by many Pods
, scheduled on different GKE nodes
. Furthermore it's really simple to configure and it works on GKE out of the box.ReadWriteMany
mode, I agree that something like NFS may be the only solution as GCE Persistent Disk doesn't provide such capability.PVC
. This step was actually already done by yourself but you got lost a bit in further steps. Let me explain how it works.annotations
section can be omitted):# Request a persistent volume for web content
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nginx-content-claim
spec:
accessModes: [ReadOnlyMany]
resources:
requests:
storage: 5Gi
accessModes
really works. In fact it's a widely misunderstood concept. First of all you cannot define access modes in PVC
in a sense of putting there the constraints you want. Supported access modes are inherent feature of a particular storage type. They are already defined by the storage provider.PVC
definition is requesting a PV
that supports the particular access mode or access modes. Note that it's in a form of a list which means you may provide many different access modes that you want your PV
to support.ReadOnlyMany
mode." You're asking this way for a storage that will satisfy your requirements. Keep in mind however that you can be given more than you ask. And this is also our scenario when asking for a PV
that supports ReadOnlyMany
mode in GCP. It creates for us a PersistentVolume
which meets our requirements we listed in accessModes
section but it also supports ReadWriteOnce
mode. Although we didn't ask for something that also supports ReadWriteOnce
you will probably agree with me that storage which has a built-in support for those two modes fully satisfies our request for something that supports ReadOnlyMany
. So basically this is the way it works.PV
that was automatically provisioned by GCP in response for your PVC
supports those two accessModes
and if you don't specify explicitely in Pod
or Deployment
definition that you want to mount it in read-only mode, by default it is mounted in read-write mode.Pod
that was able to successfully mount the PersistentVolume
:kubectl exec -ti pod-name -- /bin/bash
"Error 400: RESOURCE_IN_USE_BY_ANOTHER_RESOURCE"
node
in ReadWriteOnce
mode and it cannot be mounted by another node
on which the rest of your Pods
were scheduled.ReadOnlyMany
mode, you need to specify it explicitely in your Deployment
definition by adding readOnly: true
statement in the volumes
section under Pod's
template specification like below:apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: nginx-content
volumes:
- name: nginx-content
persistentVolumeClaim:
claimName: nginx-content-claim
readOnly: true
readOnly
mode, first we need to pre-populate such volume with data. Otherwise you'll see another error message, saying that unformatted volume cannot be mounted in read only mode.Pod
which will serve only for copying data which was already uploaded to one of our GKE nodes to our destination PV
.PersistentVolume
with data can be done in many different ways. You can mount in such Pod
only your PersistentVolume
that you will be using in your Deployment
and get your data using curl
or wget
from some external location saving it directly on your destination PV
. It's up to you.Pod
a directory
, partition
or disk
(in my example I use a directory /var/tmp/test
located on one of my GKE nodes) available on one of our kubernetes nodes. It's much more flexible solution than hostPath
as we don't have to care about scheduling such Pod
to particular node, that contains the data. Specific node affinity rule is already defined in PersistentVolume
and Pod
is automatically scheduled on specific node.StorageClass
:apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
PersistentVolume
definition:apiVersion: v1
kind: PersistentVolume
metadata:
name: example-pv
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /var/tmp/test
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- <gke-node-name>
PersistentVolumeClaim
:apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myclaim
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 10Gi
storageClassName: local-storage
Pod
which will serve only for copying data from our GKE node to our GCE Persistent Disk.apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: myfrontend
image: nginx
volumeMounts:
- mountPath: "/mnt/source"
name: mypd
- mountPath: "/mnt/destination"
name: nginx-content
volumes:
- name: mypd
persistentVolumeClaim:
claimName: myclaim
- name: nginx-content
persistentVolumeClaim:
claimName: nginx-content-claim
Pod
is only to allow us to copy our data to the destination PV
. Eventually our PV
will be mounted in completely different path.Pod
is created and both volumes are successfully mounted, we can attach to it by running:kubectl exec -ti my-pod -- /bin/bash
Pod
simply run:cp /mnt/source/* /mnt/destination/
exit
and delete our temporary Pod
:kubectl delete pod mypod
Deployment
and our PersistentVolume
finally can be mounted in readOnly
mode by all the Pods
located on various GKE nodes:apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: nginx-content
volumes:
- name: nginx-content
persistentVolumeClaim:
claimName: nginx-content-claim
readOnly: true
Pods
will be scheduled only on one particular node, you can give up on using GCE Persistent Disk at all and switch to the above mentioned local volume. This way all your Pods
will be able not only to read from it but also to write to it at the same time. The only caveat is that all those Pods
will be running on a single node.