一、Kubernetes 支持的存储类型
1、从官方手册查看
[root@master ~]# kubectl explain pod.spec.volumes
KIND: Pod
VERSION: v1
RESOURCE: volumes <[]Object>
DESCRIPTION:
List of volumes that can be mounted by containers belonging to the pod.
More info: https://kubernetes.io/docs/concepts/storage/volumes
Volume represents a named volume in a pod that may be accessed by any
container in the pod.
FIELDS:
awsElasticBlockStore <Object>
AWSElasticBlockStore represents an AWS Disk resource that is attached to a
kubelet's host machine and then exposed to the pod. More info:
https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
azureDisk <Object>
AzureDisk represents an Azure Data Disk mount on the host and bind mount to
the pod.
azureFile <Object>
AzureFile represents an Azure File Service mount on the host and bind mount
to the pod.
cephfs <Object>
CephFS represents a Ceph FS mount on the host that shares a pod's lifetime
cinder <Object>
Cinder represents a cinder volume attached and mounted on kubelets host
machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
configMap <Object>
ConfigMap represents a configMap that should populate this volume
csi <Object>
CSI (Container Storage Interface) represents storage that is handled by an
external CSI driver (Alpha feature).
downwardAPI <Object>
DownwardAPI represents downward API about the pod that should populate this
volume
emptyDir <Object>
EmptyDir represents a temporary directory that shares a pod's lifetime.
More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir
fc <Object>
FC represents a Fibre Channel resource that is attached to a kubelet's host
machine and then exposed to the pod.
flexVolume <Object>
FlexVolume represents a generic volume resource that is
provisioned/attached using an exec based plugin.
flocker <Object>
Flocker represents a Flocker volume attached to a kubelet's host machine.
This depends on the Flocker control service being running
gcePersistentDisk <Object>
GCEPersistentDisk represents a GCE Disk resource that is attached to a
kubelet's host machine and then exposed to the pod. More info:
https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
gitRepo <Object>
GitRepo represents a git repository at a particular revision. DEPRECATED:
GitRepo is deprecated. To provision a container with a git repo, mount an
EmptyDir into an InitContainer that clones the repo using git, then mount
the EmptyDir into the Pod's container.
glusterfs <Object>
Glusterfs represents a Glusterfs mount on the host that shares a pod's
lifetime. More info: https://examples.k8s.io/volumes/glusterfs/README.md
hostPath <Object>
HostPath represents a pre-existing file or directory on the host machine
that is directly exposed to the container. This is generally used for
system agents or other privileged things that are allowed to see the host
machine. Most containers will NOT need this. More info:
https://kubernetes.io/docs/concepts/storage/volumes#hostpath
iscsi <Object>
ISCSI represents an ISCSI Disk resource that is attached to a kubelet's
host machine and then exposed to the pod. More info:
https://examples.k8s.io/volumes/iscsi/README.md
name <string> -required-
Volume's name. Must be a DNS_LABEL and unique within the pod. More info:
https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
nfs <Object>
NFS represents an NFS mount on the host that shares a pod's lifetime More
info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
persistentVolumeClaim <Object>
PersistentVolumeClaimVolumeSource represents a reference to a
PersistentVolumeClaim in the same namespace. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
photonPersistentDisk <Object>
PhotonPersistentDisk represents a PhotonController persistent disk attached
and mounted on kubelets host machine
portworxVolume <Object>
PortworxVolume represents a portworx volume attached and mounted on
kubelets host machine
projected <Object>
Items for all in one resources secrets, configmaps, and downward API
quobyte <Object>
Quobyte represents a Quobyte mount on the host that shares a pod's lifetime
rbd <Object>
RBD represents a Rados Block Device mount on the host that shares a pod's
lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md
scaleIO <Object>
ScaleIO represents a ScaleIO persistent volume attached and mounted on
Kubernetes nodes.
secret <Object>
Secret represents a secret that should populate this volume. More info:
https://kubernetes.io/docs/concepts/storage/volumes#secret
storageos <Object>
StorageOS represents a StorageOS volume attached and mounted on Kubernetes
nodes.
vsphereVolume <Object>
VsphereVolume represents a vSphere volume attached and mounted on kubelets
host machine
2、重点关注
二、存储卷的使用方式
1、存储卷的配置组成
pod.spec.volumes字段定义在pod之上的存储卷列表、其支持使用多种不同类型的存储卷且配置参数差别很大
pod.spec.containers.volumeMounts字段在容器上定义的存储卷挂在列表、它只能挂在当前pod资源中定义的具体存储卷、当然、也可以不挂在任何存储卷
2、示例文档
apiVersion: v1
kind: Pod
metadata:
name: vol-emptydir-pod
spec:
volumes:
- name: html
emptyDir: {}
containers:
- name: nginx
image: nginx:1.12-alpine
volumeMounts:
- name: html
mountPath: /usr/share/nginx/html
- name: pagegen
image: alpine
volumeMounts:
- name: html
mountPath: /html
command: ["/bin/sh", "-c"]
args:
- while true; do
echo $(hostname) $(date) >> /html/index.html;
sleep 10;
done
事实上、也只有多个容器挂在同一个存储卷时"共享"才有具体的意义
2、重要字段详解
[root@master ~]# kubectl explain pod.spec.containers.volumeMounts
KIND: Pod
VERSION: v1
RESOURCE: volumeMounts <[]Object>
DESCRIPTION:
Pod volumes to mount into the container's filesystem. Cannot be updated.
VolumeMount describes a mounting of a Volume within a container.
FIELDS:
mountPath <string> -required-
#挂载点路径、容器文件系统上的路径、必选字段
Path within the container at which the volume should be mounted. Must not
contain ':'.
mountPropagation <string>
mountPropagation determines how mounts are propagated from the host to
container and the other way around. When not set, MountPropagationNone is
used. This field is beta in 1.10.
name <string> -required-
#指定要挂在的存储的名称、必选字段
This must match the Name of a Volume.
readOnly <boolean>
#是否挂在只读卷
Mounted read-only if true, read-write otherwise (false or unspecified).
Defaults to false.
subPath <string>
#挂在存储卷时使用的子路径、即mountPath指定的路径下使用一个子路径作为挂载点
Path within the volume from which the container's volume should be mounted.
Defaults to "" (volume's root).
subPathExpr <string>
Expanded path within the volume from which the container's volume should be
mounted. Behaves similarly to SubPath but environment variable references
$(VAR_NAME) are expanded using the container's environment. Defaults to ""
(volume's root). SubPathExpr and SubPath are mutually exclusive.
三、临时存储卷
1、应用场景
同一pod内的多个容器间问价你的共享、或者作为容器数据的临时存储目录用户数据缓存系统等
2、字段属性详解
[root@master ~]# kubectl explain pod.spec.volumes.emptyDir
KIND: Pod
VERSION: v1
RESOURCE: emptyDir <Object>
DESCRIPTION:
EmptyDir represents a temporary directory that shares a pod's lifetime.
More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir
Represents an empty directory for a pod. Empty directory volumes support
ownership management and SELinux relabeling.
FIELDS:
medium <string>
#medium：此目录所在的存储介质的类型、可取值为"default" 或 "Memory"、默认为default、表示使用节点的默认存储介质
"Memory"表示使用基于RAM的临时文件系统tmpfs、空间受限于内存、但性能非常好、通常用于容器中的应用提供缓存空间
What type of storage medium should back this directory. The default is ""
which means to use the node's default medium. Must be an empty string
(default) or Memory. More info:
https://kubernetes.io/docs/concepts/storage/volumes#emptydir
sizeLimit <string>
#当前存储卷的空间限额、默认值为nil、表示不限制；不过、在medium字段值为"Memory"时建议务必定义此限额
Total amount of local storage required for this EmptyDir volume. The size
limit is also applicable for memory medium. The maximum usage on memory
medium EmptyDir would be the minimum value between the SizeLimit specified
here and the sum of memory limits of all containers in a pod. The default
is nil which means that the limit is undefined. More info:
http://kubernetes.io/docs/user-guide/volumes#emptydir
3、资源清单
[root@master chapter7]# cat vol-emptydir.yaml
apiVersion: v1
kind: Pod
metadata:
name: vol-emptydir-pod
spec:
volumes:
- name: html
emptyDir: {}
containers:
- name: nginx
image: nginx:1.12-alpine
volumeMounts:
- name: html
mountPath: /usr/share/nginx/html
- name: pagegen
image: alpine
volumeMounts:
- name: html
mountPath: /html
command: ["/bin/sh", "-c"]
args:
- while true; do
echo $(hostname) $(date) >> /html/index.html;
sleep 10;
done
4、创建运行
[root@master chapter7]# kubectl apply -f vol-emptydir.yaml pod/vol-emptydir-pod created
5、查看验证
[root@master chapter7]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES vol-emptydir-pod 2/2 Running 0 14m 10.244.2.68 node2 <none> <none> [root@master chapter7]# culr 10.244.2.68 -bash: culr: command not found [root@master chapter7]# curl 10.244.2.68 vol-emptydir-pod Mon Aug 10 07:58:35 UTC 2020 vol-emptydir-pod Mon Aug 10 07:58:45 UTC 2020 vol-emptydir-pod Mon Aug 10 07:58:55 UTC 2020 ...... vol-emptydir-pod Mon Aug 10 08:17:56 UTC 2020 vol-emptydir-pod Mon Aug 10 08:18:06 UTC 2020 vol-emptydir-pod Mon Aug 10 08:18:16 UTC 2020
6、查看详细信息
pod 资源的详细信息中会显示存储卷的相关存储的相关状态、包裹其创建成功、相关的类型及参数、以及容器中的挂载状态等信息
[root@master chapter7]# kubectl describe pods vol-emptydir-pod
Name: vol-emptydir-pod
Namespace: default
Priority: 0
Node: node2/192.168.118.20
......
Containers:
nginx:
Container ID: docker://b7e816b54cb4145f47d9bb8f4375963d533966df2d3da682b3a58a30f90c19f7
......
Mounts:
/usr/share/nginx/html from html (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-pwl2t (ro)
pagegen:
......
Command:
/bin/sh
-c
Args:
while true; do echo $(hostname) $(date) >> /html/index.html; sleep 10; done
State: Running
Started: Mon, 10 Aug 2020 15:58:35 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/html from html (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-pwl2t (ro)
Conditions:
......
Volumes:
html:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
default-token-pwl2t:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-pwl2t
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 8m27s kubelet, node2 Pulling image "nginx:1.12-alpine"
Normal Scheduled 8m26s default-scheduler Successfully assigned default/vol-emptydir-pod to node2
Normal Pulled 8m10s kubelet, node2 Successfully pulled image "nginx:1.12-alpine"
Normal Created 8m10s kubelet, node2 Created container nginx
Normal Started 8m9s kubelet, node2 Started container nginx
Normal Pulling 8m9s kubelet, node2 Pulling image "alpine"
Normal Pulled 7m39s kubelet, node2 Successfully pulled image "alpine"
Normal Created 7m39s kubelet, node2 Created container pagegen
Normal Started 7m39s kubelet, node2 Started container pagegen
7、emptyDir存储卷也可以基于RAM创建tmpfs文件系统的存储卷、常用于为容器的应用提供高性能缓存
volumes:
- name: cache
emptyDir:
medium: Memory
emptyDir卷简单易用、但技能用于临时存储、另外还存在一些类型的存储卷建构在emptyDir之上、并额外提供了它所没有的功能例如gitRepo
四、临时存储卷(gitRepo)
基于emptyDir构建的gitRepo存储卷可以在pod对象的声明周期其实时从相应的GIT仓库中复制相应的数据文件到底层的 中、从而使得它具有一定意义上的持久性
gitRepo可以看做是emptyDir存储卷的一种实际应用、使用该存储卷的pod资源可以通过挂在目录访问指定的代码仓库中的数据
使用gitRepo存储卷的pod资源在创建时、会首选创建一个空目录并克隆一份指定的git仓库中的数据至该目录、而后再创建容器并挂在存储卷
1、字段详解
[root@master ~]# kubectl explain pod.spec.volumes.gitRepo
KIND: Pod
VERSION: v1
RESOURCE: gitRepo <Object>
DESCRIPTION:
GitRepo represents a git repository at a particular revision. DEPRECATED:
GitRepo is deprecated. To provision a container with a git repo, mount an
EmptyDir into an InitContainer that clones the repo using git, then mount
the EmptyDir into the Pod's container.
Represents a volume that is populated with the contents of a git
repository. Git repo volumes do not support ownership management. Git repo
volumes support SELinux relabeling. DEPRECATED: GitRepo is deprecated. To
provision a container with a git repo, mount an EmptyDir into an
InitContainer that clones the repo using git, then mount the EmptyDir into
the Pod's container.
FIELDS:
directory <string>
#目标目录名称，名称中不包含 ".." 字符 "."表示将仓库中的数据直接复制到卷目录中、否则、即为复制到卷目录中以用户指定的字符喜欢为名称的子目录中
Target directory name. Must not contain or start with '..'. If '.' is
supplied, the volume directory will be the git repository. Otherwise, if
specified, the volume will contain the git repository in the subdirectory
with the given name.
repository <string> -required-
#git仓库的URL、必选字段
Repository URL
revision <string>
#特定revision的提交哈希码
Commit hash for the specified revision
2、资源清单
[root@master chapter7]# cat vol-gitrepo.yaml
apiVersion: v1
kind: Pod
metadata:
name: vol-gitrepo
spec:
containers:
- name: nginx
image: nginx:1.12-alpine
volumeMounts:
- name: html
mountPath: /usr/share/nginx/html
volumes:
- name: html
gitRepo:
repository: https://github.com/iKubernetes/k8s_book.git
directory: .
revision: "master"
3、创建运行
[root@master chapter7]# kubectl apply -f vol-gitrepo.yaml pod/vol-gitrepo created
使用gitRepo存储卷的pod资源运行的工作节点上必须安装有git程序，否则克隆仓库的操作将无从完成
4、效果验证
[root@master chapter7]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-example 1/1 Running 1 4d23h 10.244.2.9 node2 <none> <none>
vol-emptydir-pod 2/2 Running 0 64m 10.244.2.68 node2 <none> <none>
vol-gitrepo 0/1 ContainerCreating 0 78s <none> node1 <none> <none>
[root@master chapter7]# kubectl describe pod vol-gitrepo
Name: vol-gitrepo
Namespace: default
Priority: 0
Node: node1/192.168.118.19
Start Time: Mon, 10 Aug 2020 17:00:31 +0800
Labels: <none>
Annotations: Status: Pending
IP:
IPs: <none>
Containers:
nginx:
Container ID:
Image: nginx:1.12-alpine
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from html (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-pwl2t (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
html:
Type: GitRepo (a volume that is pulled from git when the pod is created)
Repository: https://github.com/iKubernetes/k8s_book.git
Revision: master
default-token-pwl2t:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-pwl2t
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 94s default-scheduler Successfully assigned default/vol-gitrepo to node1
Warning FailedMount 25s (x7 over 88s) kubelet, node1 MountVolume.SetUp failed for volume "html" : failed to exec 'git clone -- https://github.com/iKubernetes/k8s_book.git .': Cloning into '.'...
fatal: could not read Username for 'https://github.com': No such device or address
: exit status 128
kubernetes 1.12版本起、gitRepo存储卷已经被废弃、所以在之后的版本中若要使用它配置pod对象、建议读者借助初始化容器将仓库中的数据复制到emptyDir存储卷上，并在主容器中使用此存储卷
五、临时存储卷(hostPath)
hostPath类型的存储卷时指将工作节点上某个文件系统的目录或文件挂在于pod中的一种存储卷、它可以独立于pod资源的生命周期、因而具有持久性
但它是工作节点本地的存储空间、仅使用于特定情况下的存储卷使用需要
例如：将工作节点陕干的文件系统关联为pod的存储卷、从而使得容器访问节点文件系统上的数据、这一点在运行由管理任务的系统级pod资源需要访问节点上的文件时尤为有用
1、字段详解
DirectoryOrCreate #指定的路径不存在时定将其创建为0755的空目录。属主属组均为Kubelet If nothing exists at the given path, an empty directory will be created there as needed with permission set to 0755, having the same group and ownership with Kubelet. Directory：A directory must exist at the given path #必须存在的目录路径 FileOrCreate： #指定的路径不存在时自动创建其创建权限为0644的空文件、属主和属组同时Kubelet If nothing exists at the given path, an empty file will be created there as needed with permission set to 0644, having the same group and ownership with Kubelet. File：A file must exist at the given path #必须存在的文件路径 Socket：A UNIX socket must exist at the given path #必须存在的socket文件路径 CharDevic：A character device must exist at the given path #必须存在的字符设备文件路径 BlockDevice：A block device must exist at the given path #不需存在的块设备文件路径
2、redis环境
1、资源清单
[root@master chapter5]# cat redis.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
labels:
app: default
spec:
replicas: 1
selector:
matchLabels:
app: redis
role: logstor
template:
metadata:
labels:
app: redis
role: logstor
spec:
containers:
- name: redis
image: redis:4.0-alpine
ports:
- name: redis
containerPort: 63
2、创建运行
[root@master chapter5]# kubectl apply -f redis.yaml deployment.apps/redis created [root@master chapter7]# kubectl expose deployment redis --port=6379 service/redis exposed
3、验证效果
[root@master chapter7]# kubectl expose deployment redis --port=6379 service/redis exposed [root@master chapter7]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6d18h redis ClusterIP 10.106.30.154 <none> 6379/TCP 51s [root@master chapter5]# kubectl get all -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/redis-588694bf8c-s6tls 1/1 Running 0 54s 10.244.1.42 node1 <none> <none> [root@master chapter7]# kubectl exec -it pod/redis-588694bf8c-s6tls -- /bin/sh /data # nslookup redis.default.svc.cluster.local Server: 10.96.0.10 Address: 10.96.0.10:53 Name: redis.default.svc.cluster.local Address: 10.106.30.154 /data # ls dump.rdb /data # redis-cli -h redis.default.svc.cluster.local redis.default.svc.cluster.local:6379> keys * (empty list or set)
3、hostPath环境
1、资源清单
[root@master chapter7]# cat vol-hostpath.yaml
apiVersion: v1
kind: Pod
metadata:
name: vol-hostpath-pod
spec:
containers:
- name: filebeat
image: ikubernetes/filebeat:5.6.7-alpine
env:
- name: REDIS_HOST
value: redis.default.svc.cluster.local
- name: LOG_LEVEL
value: info
volumeMounts:
- name: varlog
mountPath: /var/log
- name: socket
mountPath: /var/run/docker.sock
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: socket
hostPath:
path: /var/run/docker.sock
2、创建运行
[root@master chapter7]# kubectl apply -f vol-hostpath.yaml pod/vol-hostpath-pod created [root@master ~]# kubectl get pods NAME READY STATUS RESTARTS AGE redis-588694bf8c-s6tls 1/1 Running 0 16h vol-hostpath-pod 1/1 Running 0 15h
3、验证效果
[root@master ~]# kubectl exec -it vol-hostpath-pod -- /bin/sh
/ # ps aux
PID USER TIME COMMAND
1 root 0:19 /usr/local/bin/filebeat -e -c /etc/filebeat/filebeat.yml
14 root 0:00 /bin/sh
20 root 0:00 ps aux
/ # cat /etc/filebeat/filebeat.yml
filebeat.registry_file: /var/log/containers/filebeat_registry
filebeat.idle_timeout: 5s
filebeat.spool_size: 2048
logging.level: info
filebeat.prospectors:
- input_type: log
paths:
- "/var/log/containers/*.log"
- "/var/log/docker/containers/*.log"
- "/var/log/startupscript.log"
- "/var/log/kubelet.log"
- "/var/log/kube-proxy.log"
- "/var/log/kube-apiserver.log"
- "/var/log/kube-controller-manager.log"
- "/var/log/kube-scheduler.log"
- "/var/log/rescheduler.log"
- "/var/log/glbc.log"
- "/var/log/cluster-autoscaler.log"
symlinks: true
json.message_key: log
json.keys_under_root: true
json.add_error_key: true
multiline.pattern: '^s'
multiline.match: after
document_type: kube-logs
tail_files: true
fields_under_root: true
output.redis:
hosts: ${REDIS_HOST:?No Redis host configured. Use env var REDIS_HOST to set host.}
key: "filebeat"
/ # nslookup redis.default.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve
Name: redis.default.svc.cluster.local
Address 1: 10.106.30.154 redis.default.svc.cluster.local