PersistentVolume（PV） 数据持久卷
对存储资源的创建与使用进行抽象，让存储可以做为集群的资源进行管理
PersistentVolumeClaim(PVC) 数据持久卷请求
让用户不需要关心具体的volume实现细节
静态PV的实现：
Kubernetes支持持久卷的存储插件： https://kubernetes.io/docs/concepts/storage/persistent-volumes/
首先配置PV：
1 #在/ifs/kubernetes/目录下创建pv01,pv02,pv03 2 [root@node2 ~]# cd /ifs/kubernetes/ 3 [root@node2 kubernetes]# mkdir pv01 4 [root@node2 kubernetes]# mkdir pv02 5 [root@node2 kubernetes]# mkdir pv03 6 7 #通过pv.yaml文件创建三个PV 8 #访问模式： 9 #ReadWriteMany RWX 所有pod读写，应用场景：数据独立，块设备 10 #ReadWriteOnce RWO 单个Pod读写 11 #ReadOnlyMany ROX 所有pod只读 应用场景：数据共享，文件系统 12 [root@master ~]# cat pv.yaml 13 apiVersion: v1 14 kind: PersistentVolume 15 metadata: 16 name: pv01 #PV的名字 17 spec: 18 capacity: #定义容量为5G 19 storage: 5Gi 20 accessModes: #访问模式为ReadWriteMany 21 - ReadWriteMany 22 nfs: #卷的类型为nfs 23 path: /ifs/kubernetes/pv01 #路径为/ifs/kubernetes/pv01 24 server: 192.168.1.63 #nfs的服务器为：192.168.1.63 25 --- 26 apiVersion: v1 27 kind: PersistentVolume 28 metadata: 29 name: pv02 30 spec: 31 capacity: 32 storage: 10Gi 33 accessModes: 34 - ReadWriteMany 35 nfs: 36 path: /ifs/kubernetes/pv02 37 server: 192.168.1.63 38 --- 39 apiVersion: v1 40 kind: PersistentVolume 41 metadata: 42 name: pv03 43 spec: 44 capacity: 45 storage: 20Gi 46 accessModes: 47 - ReadWriteMany 48 nfs: 49 path: /ifs/kubernetes/pv03 50 server: 192.168.1.63 51 52 [root@master ~]# 53 [root@master ~]# kubectl apply -f pv.yaml 54 persistentvolume/pv01 created 55 persistentvolume/pv02 created 56 persistentvolume/pv03 created 57 [root@master ~]# kubectl get pv 58 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE 59 pv01 5Gi RWX Retain Available 11s 60 pv02 10Gi RWX Retain Available 11s 61 pv03 20Gi RWX Retain Available 11s 62 [root@master ~]#
以deployment的方式使用静态PVC
1 [root@master ~]# #通过deployment控制器启动一个Pod,在这个Pod中数据 2 #持久化通过PVC来实现 3 #deployment和PVC通过name:my-pvc进行关联 4 #两个yaml文件通常是写在一起的 5 [root@master ~]# cat deployment2.yaml 6 apiVersion: apps/v1 7 kind: Deployment 8 metadata: 9 labels: 10 app: web 11 name: web 12 spec: 13 replicas: 1 14 selector: 15 matchLabels: 16 app: web 17 strategy: {} 18 template: 19 metadata: 20 labels: 21 app: web 22 spec: 23 containers: 24 - image: nginx 25 name: nginx 26 resources: {} 27 volumeMounts: 28 - name: data 29 mountPath: /usr/share/nginx/html 30 31 volumes: 32 - name: data 33 persistentVloumeClaim: 34 claimName: my-pvc 35 --- 36 apiVersion: v1 37 38 kind: PersistentVolumeClaim 39 metadata: 40 name: my-pvc 41 spec: 42 accessModes: 43 - ReadWriteMany 44 resources: 45 requests: 46 storage: 8Gi 47 [root@master ~]# 48 49 [root@master ~]# kubectl apply -f deployment2.yaml 50 deployment.apps/web created 51 persistentvolumeclaim/my-pvc created 52 [root@master ~]# kubectl get pod 53 NAME READY STATUS RESTARTS AGE 54 web-7d74df4646-fr88j 1/1 Running 0 15s 55 [root@master ~]# kubectl get pvc 56 NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE 57 my-pvc Bound pv02 10Gi RWX 27s 58 [root@master ~]#
PVC与PV的关系
PV与PVC是一一对应的，名为my-pvc的PVC最终匹配的是PV02,一个PVC只可能匹配一个pv
PVC与PV之间的对应是由匹配条件决定的，PVC与PV的匹配条件是：
1 容量
2 访问模式
3 标签
1 [root@master ~]# kubectl get pv 2 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE 3 pv01 5Gi RWX Retain Available 67s 4 pv02 10Gi RWX Retain Bound default/my-pvc 67s 5 pv03 20Gi RWX Retain Available 67s 6 [root@master ~]# #PVX删除了之后，对应的PV中的数据并不会删除 7 #如下图中的pv02，my-pvc删除后，pv02中的数据还是存在的可以对 8 #pv02中的数据进行删除或者备份 9 #同时pv02已不可以再次使用，即它的状态无法再回到available 10 #如果确定不再使用pv02中的数据，可以手动删除pv02 11 #pv删除之后对就的存储是否删除是由RECLAIM POLICY决定的： 12 #RECLAIM POLICY的状态有 13 #Delete: 直接删除PV+数据 （不推荐） 14 #Recycle: 清除数据，但保留PV（被废弃） 15 #Retain: 保留PV&数据 （推荐） 16 17 [root@master ~]# kubectl delete -f deployment2.yaml 18 deployment.apps "web" deleted 19 persistentvolumeclaim "my-pvc" deleted 20 [root@master ~]# kubectl get pv,pvc 21 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE 22 persistentvolume/pv01 5Gi RWX Retain Available 86m 23 persistentvolume/pv02 10Gi RWX Retain Released default/my-pvc 86m 24 persistentvolume/pv03 20Gi RWX Retain Available 86m 25 [root@master ~]# 26
动态PVC的实现：
静态PVC的缺点：
PV需要提前手动创建，PVC在使用PV时可能会出现实现用的PV大于PVC请求的PV
动态PVC可以解决这个问题，当PVC需要PV时会自动创建PVC需求的PV
StorageClass用于自动创建PV, 并不是所有存储类型都支持StorageClass的自动创建，
查看StorageClass支持的存储列表：
https://kubernetes.io/docs/concepts/storage/storage-classes/
如果默认不支持的存储类型仍需要使用动态PVC，那么可以使用的插件列表如下
https://github.com/kubernetes-retired/external-storage/
如下示例，以nfs的存储类型配置动态PVC
StorageClass默认不是支持nfs类型的PVC的，所以需要手动安装插件
1 [root@master NFS]# rz -E 2 rz waiting to receive. 3 [root@master NFS]# ls 4 nfs-client.zip 5 [root@master NFS]# unzip nfs-client.zip 6 [root@master NFS]# cd nfs-client/ 7 [root@master nfs-client]# ls 8 class.yaml deployment.yaml rbac.yaml 9 [root@master nfs-client]# #定义一下StorageClass资源 10 [root@master nfs-client]# cat class.yaml 11 apiVersion: storage.k8s.io/v1 12 kind: StorageClass 13 metadata: 14 name: managed-nfs-storage #storageclass资源的名字 15 provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME' 16 parameters: 17 archiveOnDelete: "true" 18 [root@master nfs-client]# #以容器的方式运行应用程序 19 [root@master nfs-client]# cat deployment.yaml 20 apiVersion: v1 21 kind: ServiceAccount 22 metadata: 23 name: nfs-client-provisioner 24 --- 25 kind: Deployment 26 apiVersion: apps/v1 27 metadata: 28 name: nfs-client-provisioner 29 spec: 30 replicas: 1 31 strategy: 32 type: Recreate 33 selector: 34 matchLabels: 35 app: nfs-client-provisioner 36 template: 37 metadata: 38 labels: 39 app: nfs-client-provisioner 40 spec: 41 serviceAccountName: nfs-client-provisioner 42 containers: 43 - name: nfs-client-provisioner 44 image: quay.io/external_storage/nfs-client-provisioner:latest 45 volumeMounts: 46 - name: nfs-client-root 47 mountPath: /persistentvolumes 48 env: 49 - name: PROVISIONER_NAME 50 value: fuseim.pri/ifs #这个变量要与class.yaml文件中的一致 51 - name: NFS_SERVER 52 value: 192.168.1.63 #nfs服务器的地址 53 - name: NFS_PATH 54 value: /ifs/kubernetes 55 volumes: 56 - name: nfs-client-root 57 nfs: 58 server: 192.168.1.63 59 path: /ifs/kubernetes 60 [root@master nfs-client]# #rbac.yaml的作用是授权nfs-client-provisioner这个容器中的 61 #应用程序可以访问K8S的API，从API中获取是否有新的PVC创建 62 #并根据PVC的请求自动创建PV 63 [root@master nfs-client]# cat rbac.yaml 64 kind: ServiceAccount 65 apiVersion: v1 66 metadata: 67 name: nfs-client-provisioner 68 --- 69 kind: ClusterRole 70 apiVersion: rbac.authorization.k8s.io/v1 71 metadata: 72 name: nfs-client-provisioner-runner 73 rules: 74 - apiGroups: [""] 75 resources: ["persistentvolumes"] 76 verbs: ["get", "list", "watch", "create", "delete"] 77 - apiGroups: [""] 78 resources: ["persistentvolumeclaims"] 79 verbs: ["get", "list", "watch", "update"] 80 - apiGroups: ["storage.k8s.io"] 81 resources: ["storageclasses"] 82 verbs: ["get", "list", "watch"] 83 - apiGroups: [""] 84 resources: ["events"] 85 verbs: ["create", "update", "patch"] 86 --- 87 kind: ClusterRoleBinding 88 apiVersion: rbac.authorization.k8s.io/v1 89 metadata: 90 name: run-nfs-client-provisioner 91 subjects: 92 - kind: ServiceAccount 93 name: nfs-client-provisioner 94 namespace: default 95 roleRef: 96 kind: ClusterRole 97 name: nfs-client-provisioner-runner 98 apiGroup: rbac.authorization.k8s.io 99 --- 100 kind: Role 101 apiVersion: rbac.authorization.k8s.io/v1 102 metadata: 103 name: leader-locking-nfs-client-provisioner 104 rules: 105 - apiGroups: [""] 106 resources: ["endpoints"] 107 verbs: ["get", "list", "watch", "create", "update", "patch"] 108 --- 109 kind: RoleBinding 110 apiVersion: rbac.authorization.k8s.io/v1 111 metadata: 112 name: leader-locking-nfs-client-provisioner 113 subjects: 114 - kind: ServiceAccount 115 name: nfs-client-provisioner 116 # replace with namespace where provisioner is deployed 117 namespace: default 118 roleRef: 119 kind: Role 120 name: leader-locking-nfs-client-provisioner 121 apiGroup: rbac.authorization.k8s.io 122 [root@master nfs-client]# 123 [root@master nfs-client]# kubectl apply -f . 124 storageclass.storage.k8s.io/managed-nfs-storage created 125 serviceaccount/nfs-client-provisioner created 126 deployment.apps/nfs-client-provisioner created 127 serviceaccount/nfs-client-provisioner unchanged 128 clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created 129 clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created 130 role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created 131 rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created 132 [root@master nfs-client]# kubectl get pod 133 NAME READY STATUS RESTARTS AGE 134 nfs-client-provisioner-7676dc9cfc-j4vgl 1/1 Running 0 22s 135 [root@master nfs-client]#
如上图示，PVC请求的存储类型是官方默念支持的存储类型，那么是不需要managed-nfs-storage 。
managed-nfs-storage 也正是deployment.yaml 文件容器中的应用程序
1 [root@master nfs-client]# kubectl get sc 2 NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE 3 managed-nfs-storage fuseim.pri/ifs Delete Immediate false 53m 4 [root@master nfs-client]#
通过控制器deployment3.yaml来实现动态PVC
1 [root@master ~]# vim deployment3.yaml 2 apiVersion: apps/v1 3 kind: Deployment 4 metadata: 5 labels: 6 app: web 7 name: web 8 spec: 9 name: web 10 spec: 11 replicas: 1 12 selector: 13 matchLabels: 14 app: web 15 strategy: {} 16 template: 17 metadata: 18 labels: 19 app: web 20 spec: 21 containers: 22 - image: nginx 23 name: nginx 24 resources: {} 25 volumeMounts: 26 - name: data 27 mountPath: /usr/share/nginx/html 28 29 volumes: 30 - name: data 31 persistentVolumeClaim: 32 claimName: my-pvc2 33 --- 34 apiVersion: v1 35 kind: PersistentVolumeClaim 36 metadata: 37 name: my-pvc2 38 spec: 39 storageClassName: "managed-nfs-storage" #这里的storageClassName要与上面kubectl get sc 查询出来的结果一致 40 accessModes: 41 - ReadWriteMany 42 resources: 43 requests: 44 storage: 9Gi 45 #创建pod和动态PVC 46 [root@master ~]# kubectl apply -f deployment3.yaml 47 deployment.apps/web created 48 persistentvolumeclaim/my-pvc2 created 49 [root@master ~]# 50 [root@master ~]# kubectl get pod 51 NAME READY STATUS RESTARTS AGE 52 nfs-client-provisioner-7676dc9cfc-j4vgl 1/1 Running 1 17h 53 web-748845d84d-fcclv 1/1 Running 0 54s 54 [root@master ~]# 55 [root@master ~]# kubectl get pv,pvc #容量为9G的PV会被自动创建 56 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE 57 persistentvolume/pv01 5Gi RWX Retain Available 22h 58 persistentvolume/pv02 10Gi RWX Retain Released default/my-pvc 22h 59 persistentvolume/pv03 20Gi RWX Retain Available 22h 60 persistentvolume/pvc-cfa1317b-67eb-4e36-8c4d-75e7c075e2a9 9Gi RWX Delete Bound default/my-pvc2 managed-nfs-storage 28s 61 62 NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE 63 persistentvolumeclaim/my-pvc2 Bound pvc-cfa1317b-67eb-4e36-8c4d-75e7c075e2a9 9Gi RWX managed-nfs-storage 28s 64 [root@master ~]# 65 #在NFS的物理目录下会自动创建目录default-my-pvc2-pvc-cfa1317b-67eb-4e36-8c4d-75e7c075e2a9 66 [root@node2 ~]# cd /ifs/kubernetes/ 67 [root@node2 kubernetes]# ls 68 a.txt default-my-pvc2-pvc-cfa1317b-67eb-4e36-8c4d-75e7c075e2a9 index.html pv01 pv02 pv03 69 [root@node2 kubernetes]#
验证NFS的共享目录
1 #在容器中创建一个文件验证数据是否会被持久化到这个目录中 2 [root@master ~]# kubectl exec -it web-748845d84d-fcclv -- bash 3 root@web-748845d84d-fcclv:/# cd /usr/share/nginx/html/ 4 root@web-748845d84d-fcclv:/usr/share/nginx/html# touch abc.txt 5 root@web-748845d84d-fcclv:/usr/share/nginx/html# ls 6 abc.txt 7 root@web-748845d84d-fcclv:/usr/share/nginx/html# 8 [root@node2 kubernetes]# cd default-my-pvc2-pvc-cfa1317b-67eb-4e36-8c4d-75e7c075e2a9/ 9 [root@node2 default-my-pvc2-pvc-cfa1317b-67eb-4e36-8c4d-75e7c075e2a9]# ls 10 abc.txt 11 [root@node2 default-my-pvc2-pvc-cfa1317b-67eb-4e36-8c4d-75e7c075e2a9]# 12 13 #为控制器创建3个副本，进入其它副本查看是否实现了数据共享 14 [root@master ~]# kubectl scale deploy web --replicas=3 15 deployment.apps/web scaled 16 [root@master ~]# kubectl get pod 17 NAME READY STATUS RESTARTS AGE 18 nfs-client-provisioner-7676dc9cfc-j4vgl 1/1 Running 1 17h 19 web-748845d84d-87dh8 1/1 Running 0 9s 20 web-748845d84d-fcclv 1/1 Running 0 32m 21 web-748845d84d-pmbdf 1/1 Running 0 9s 22 [root@master ~]# #进入另外一个Pod中查看 23 [root@master ~]# kubectl exec -it web-748845d84d-pmbdf -- bash 24 root@web-748845d84d-pmbdf:/# cd /usr/share/nginx/html/ 25 root@web-748845d84d-pmbdf:/usr/share/nginx/html# ls 26 abc.txt 27 root@web-748845d84d-pmbdf:/usr/share/nginx/html#
验证动态PVC 对PV的回收策略
1 #通过控制器删除Pod查看持久化在外部的数据 2 #pod删除后，对应的pv,pvc都被删除了 3 [root@master ~]# kubectl delete -f deployment3.yaml 4 deployment.apps "web" deleted 5 persistentvolumeclaim "my-pvc2" deleted 6 [root@master ~]# 7 [root@master ~]# kubectl get pod 8 NAME READY STATUS RESTARTS AGE 9 nfs-client-provisioner-7676dc9cfc-j4vgl 1/1 Running 1 18h 10 [root@master ~]# 11 [root@master ~]# kubectl get pv,pvc 12 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE 13 persistentvolume/pv01 5Gi RWX Retain Available 23h 14 persistentvolume/pv02 10Gi RWX Retain Released default/my-pvc 23h 15 persistentvolume/pv03 20Gi RWX Retain Available 23h 16 [root@master ~]# 17 #持久化在外部的数据目录 default-my-pvc2-pvc-cfa1317b-67eb-4e36-8c4d-75e7c075e2a9/ 18 #会被自动归档到archived-default-my-pvc2-pvc-cfa1317b-67eb-4e36-8c4d-75e7c075e2a9目录 19 [root@node2 kubernetes]# ls 20 archived-default-my-pvc2-pvc-cfa1317b-67eb-4e36-8c4d-75e7c075e2a9 a.txt index.html pv01 pv02 pv03 21 [root@node2 kubernetes]#