1、K8s为什么要使用存储？
答：k8s中的副本控制器保证了pod的始终存储，却保证不了Pod中的数据。只有启动一个新pod的，之前pod中的数据会随着容器的删掉而丢失。k8s中的rc启动指定数量的Pod，当某个Pod死掉了，会在新的节点启动新的Pod，k8s中想要实现数据持久化，需要使用一个叫做共享存储的，让Pod里面的数据挂载到这个共享存储上面，就算在新的节点启动新的Pod，依然可以保证数据不丢失。
2、k8s中的PV和PVC的概念。
答：PersistentVolume(简称为PV，持久化存储)，由管理员添加的一个存储的描述，是一个全局资源，没有namespace的限制，包含存储的类型，存储的大小和访问模式等等。它的生命周期独立于Pod，例如当使用它的Pod销毁时对PV没有影响。
PersistentVolumeClaim(简称为PVC)，是Namespace里面的资源，描述对PV的一个请求。请求信息包含存储大小，访问模式等等。注意，PV和PVC是一一绑定的。
3、首先这里创建PV使用到的是NFS，这里首先安装一下NFS，如下所示：
需要Master主节点先安装NFS服务端，在所有Node节点安装NFS客户端，不然Pod无法使用，因为挂载不上，在三台机器都执行下面的命令即可。
1 [root@k8s-master ~]# yum install nfs-utils.x86_64 2 Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager 3 4 This system is not registered with an entitlement server. You can use subscription-manager to register. 5 6 Loading mirror speeds from cached hostfile 7 * base: mirrors.tuna.tsinghua.edu.cn 8 * extras: mirrors.bfsu.edu.cn 9 * updates: mirrors.bfsu.edu.cn 10 base | 3.6 kB 00:00:00 11 extras | 2.9 kB 00:00:00 12 updates | 2.9 kB 00:00:00 13 Resolving Dependencies 14 --> Running transaction check 15 ---> Package nfs-utils.x86_64 1:1.3.0-0.61.el7 will be updated 16 ---> Package nfs-utils.x86_64 1:1.3.0-0.66.el7 will be an update 17 --> Finished Dependency Resolution 18 19 Dependencies Resolved 20 21 ================================================================================================================================================================================================================= 22 Package Arch Version Repository Size 23 ================================================================================================================================================================================================================= 24 Updating: 25 nfs-utils x86_64 1:1.3.0-0.66.el7 base 412 k 26 27 Transaction Summary 28 ================================================================================================================================================================================================================= 29 Upgrade 1 Package 30 31 Total size: 412 k 32 Is this ok [y/d/N]: y 33 Downloading packages: 34 Running transaction check 35 Running transaction test 36 Transaction test succeeded 37 Running transaction 38 Updating : 1:nfs-utils-1.3.0-0.66.el7.x86_64 1/2 39 Cleanup : 1:nfs-utils-1.3.0-0.61.el7.x86_64 2/2 40 Verifying : 1:nfs-utils-1.3.0-0.66.el7.x86_64 1/2 41 Verifying : 1:nfs-utils-1.3.0-0.61.el7.x86_64 2/2 42 43 Updated: 44 nfs-utils.x86_64 1:1.3.0-0.66.el7 45 46 Complete! 47 [root@k8s-master ~]#
在三台机器都要安装nfs-utils的包的，然后在服务器端，开始配置配置文件/etc/exports，如下所示：
1 [root@k8s-master ~]# vim /etc/exports
配置内容，如下所示：
1 # 共享data目录，允许192.168.110.*访问，即允许110段ip地址访问。读写权限，同步，不做root用户、所有用户的UID映射 2 /data 192.168.110.0/24(rw,async,no_root_squash,no_all_squash)
创建/data/目录下面的一个目录，如下所示：
1 [root@k8s-master ~]# cat /etc/exports 2 # 共享data目录，允许192.168.110.*访问，即允许110段ip地址访问。读写权限，同步，不做root用户、所有用户的UID映射 3 /data 192.168.110.0/24(rw,async,no_root_squash,no_all_squash) 4 5 6 [root@k8s-master ~]# mkdir /data/k8s 7 mkdir: cannot create directory ‘/data/k8s’: No such file or directory 8 [root@k8s-master ~]# mkdir /data/k8s -p 9 [root@k8s-master ~]#
然后重启nfs、rpcbind，如下所示：
1 [root@k8s-master ~]# systemctl restart rpcbind.service 2 [root@k8s-master ~]# systemctl restart nfs.service 3 [root@k8s-master ~]#
在另外两个节点查看是否可以查看到nfs，如下所示：
1 [root@k8s-node2 ~]# showmount -e 192.168.110.133 2 Export list for 192.168.110.133: 3 /data 192.168.110.0/24 4 [root@k8s-node2 ~]# 5 6 7 [root@k8s-node3 ~]# showmount -e 192.168.110.133 8 Export list for 192.168.110.133: 9 /data 192.168.110.0/24 10 [root@k8s-node3 ~]#
下面开始创建一个PV，如下所示：
1 [root@k8s-master ~]# cd k8s/ 2 [root@k8s-master k8s]# ls 3 book-master.war dashboard dashboard.zip deploy health heapster hpa metrics namespace pod rc skydns skydns.zip svc tomcat_demo tomcat_demo.zip 4 [root@k8s-master k8s]# mkdir volume 5 [root@k8s-master k8s]# cd volume/ 6 [root@k8s-master volume]# vim test-pv.yaml
配置内容，如下所示：
1 apiVersion: v1 2 kind: PersistentVolume 3 metadata: 4 name: testpv 5 labels: 6 type: testpv 7 spec: 8 capacity: 9 storage: 10Gi 10 accessModes: 11 - ReadWriteMany 12 persistentVolumeReclaimPolicy: Recycle 13 nfs: 14 path: "/data/k8s" 15 server: 192.168.110.133 16 readOnly: false
开始创建这个PV，并进行查看，如下所示：
1 [root@k8s-master volume]# kubectl create -f test-pv.yaml 2 persistentvolume "testpv" created 3 [root@k8s-master volume]# kubectl get pv -o wide 4 NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE 5 testpv 10Gi RWX Recycle Available 10s 6 [root@k8s-master volume]#
再创建一个5Gi的PV，修改一下配置文件，然后进行创建，名称必须不一样，标签可以一样的，如下所示：
1 [root@k8s-master volume]# vim test-pv.yaml 2 [root@k8s-master volume]# kubectl create -f test-pv.yaml 3 persistentvolume "testpv2" created 4 [root@k8s-master volume]# kubectl get pv -o wide 5 NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE 6 testpv 10Gi RWX Recycle Available 3m 7 testpv2 5Gi RWX Recycle Available 3s 8 [root@k8s-master volume]#
开始创建PVC，如下所示：
1 apiVersion: v1 2 kind: PersistentVolumeClaim 3 kind: PersistentVolumeClaim 4 apiVersion: v1 5 metadata: 6 name: nfs 7 spec: 8 accessModes: 9 - ReadWriteMany 10 resources: 11 requests: 12 storage: 1Gi
1 [root@k8s-master volume]# kubectl create -f test-pvc.yaml 2 persistentvolumeclaim "nfs" created 3 [root@k8s-master volume]# kubectl get pvc -o wide 4 NAME STATUS VOLUME CAPACITY ACCESSMODES AGE 5 nfs Bound testpv2 5Gi RWX 9s 6 [root@k8s-master volume]#
此时，可以发现pvc已经绑定到了testpv2这个PV上面了，pvc优先绑定到容量小的pv上面的。自己也可以再创建一个6G的PV，然后再创建一个7G的PVC，可以发现这个PVC会绑定到10G的这个PV上面了的。
4、K8s持久化实战，自己可以参考一下这篇https://www.cnblogs.com/wangxu01/articles/11411113.html。
这里首先引用一下https://www.cnblogs.com/biehongli/p/13150609.html，使用k8s运行Java Web项目，这里紧接着这个链接的操作，创建一个mysql，如下所示：
1 [root@k8s-master tomcat_demo]# kubectl create -f mysql-rc.yml 2 replicationcontroller "mysql" created 3 [root@k8s-master tomcat_demo]# kubectl create -f mysql-svc.yml 4 service "mysql" created 5 [root@k8s-master tomcat_demo]# kubectl get all -o wide 6 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 7 rc/mysql 1 1 1 14s mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 8 9 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 10 svc/kubernetes 10.254.0.1 <none> 443/TCP 24d <none> 11 svc/mysql 10.254.234.52 <none> 3306/TCP 11s app=mysql 12 13 NAME READY STATUS RESTARTS AGE IP NODE 14 po/mysql-4j7qk 1/1 Running 0 14s 172.16.16.3 k8s-node3 15 [root@k8s-master tomcat_demo]#
下面是mysql-rc.yaml的配置文件，如下所示：
1 [root@k8s-master tomcat_demo]# cat mysql-rc.yml 2 apiVersion: v1 3 kind: ReplicationController 4 metadata: 5 name: mysql 6 spec: 7 replicas: 1 8 selector: 9 app: mysql 10 template: 11 metadata: 12 labels: 13 app: mysql 14 spec: 15 containers: 16 - name: mysql 17 image: 192.168.110.133:5000/mysql:5.7.30 18 ports: 19 - containerPort: 3306 20 env: 21 - name: MYSQL_ROOT_PASSWORD 22 value: '123456' 23 [root@k8s-master tomcat_demo]#
下面是mysql-svc.yaml的配置文件，如下所示：
1 [root@k8s-master tomcat_demo]# cat mysql-svc.yml 2 apiVersion: v1 3 kind: Service 4 metadata: 5 name: mysql 6 spec: 7 ports: 8 - port: 3306 9 targetPort: 3306 10 selector: 11 app: mysql 12 [root@k8s-master tomcat_demo]#
下面是tomcat-rc.yml的配置文件，如下所示：
1 [root@k8s-master tomcat_demo]# kubectl get all -o wide 2 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 3 rc/mysql 1 1 1 2h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 4 rc/myweb 1 1 1 11m myweb 192.168.110.133:5000/tomcat-book:latest app=myweb 5 6 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 7 svc/kubernetes 10.254.0.1 <none> 443/TCP 24d <none> 8 svc/mysql 10.254.234.52 <none> 3306/TCP 2h app=mysql 9 svc/myweb 10.254.14.249 <nodes> 8080:30008/TCP 10m app=myweb 10 11 NAME READY STATUS RESTARTS AGE IP NODE 12 po/mysql-4j7qk 1/1 Running 0 2h 172.16.16.3 k8s-node3 13 po/myweb-f2dqn 1/1 Running 0 11m 172.16.16.5 k8s-node3 14 [root@k8s-master tomcat_demo]# vim tomcat-rc.yml 15 [root@k8s-master tomcat_demo]# cat tomcat-rc.yml 16 apiVersion: v1 17 kind: ReplicationController 18 metadata: 19 name: myweb 20 spec: 21 replicas: 1 22 selector: 23 app: myweb 24 template: 25 metadata: 26 labels: 27 app: myweb 28 spec: 29 containers: 30 - name: myweb 31 # image: 192.168.110.133:5000/tomcat:latest 32 image: 192.168.110.133:5000/tomcat-book:latest 33 ports: 34 - containerPort: 8080 35 env: 36 - name: MYSQL_SERVICE_HOST 37 # value: '192.168.110.133' # 如果已经安装了dns服务，那么这里可以直接使用mysql的svc即Service的名称mysql 38 value: '10.254.234.52' 39 - name: MYSQL_SERVICE_PORT 40 value: '3306' 41 [root@k8s-master tomcat_demo]#
下面是tomcat-svc.yml的配置文件，如下所示：
1 [root@k8s-master tomcat_demo]# cat tomcat-svc.yml 2 apiVersion: v1 3 kind: Service 4 metadata: 5 name: myweb 6 spec: 7 type: NodePort 8 ports: 9 - port: 8080 10 nodePort: 30008 11 selector: 12 app: myweb 13 [root@k8s-master tomcat_demo]#
然后可以通过Navicat连上这个机器的Mysql，然后创建一个数据库，将sql执行一下，就完成了Mysql的任务了，下面就是将web项目搞到tomcat里面，然后使用k8s启动tomcat就行了。
怎么创建java web端的镜像呢，首先镜像要运行在tomcat里面的，创建镜像的第一步，说明继承自那个基础镜像，因为程序是Java web程序，肯定是需要用到tomcat的，先将tomcat的镜像下载下来。
1 [root@k8s-master tomcat_demo]# vim Dockerfile
配置内容，如下所示：
1 # 第一步，说明继承那个基础镜像，tomcat作为基础镜像。此处告诉Docker我要做自己的镜像，以tomcat镜像为起点。 2 from docker.io/tomcat:latest3 4 # 第二步，描述镜像的所有者，可以省略。 5 MAINTAINER biehl biehl@qq.com 6 7 # 第三步，将自己的web应用放到镜像里面，将本地的war包拷贝到tomcat的webapps目录下面。可以看看程序是否可以随着tomcat启动起来。 8 COPY book-master.war /usr/local/tomcat/webapps
dokcer build命令就是用来执行Dockerfile里面所描述的每一件事情，最终会把docker镜像给我们构建出来。
1 [root@k8s-master tomcat_demo]# docker build -t tomcat-book:latest . 2 Sending build context to Docker daemon 9.699 MB 3 Step 1/3 : FROM 192.168.110.133:5000/tomcat:latest 4 ---> 2eb5a120304e 5 Step 2/3 : MAINTAINER biehl biehl@qq.com 6 ---> Running in a51d06f7bddb 7 ---> 0be82c0d264e 8 Removing intermediate container a51d06f7bddb 9 Step 3/3 : COPY book-master.war /usr/local/tomcat/webapps 10 ---> 98fbba0fd03c 11 Removing intermediate container e71c8a9f14c9 12 Successfully built 98fbba0fd03c 13 [root@k8s-master tomcat_demo]# docker images 14 REPOSITORY TAG IMAGE ID CREATED SIZE 15 book-master.war latest 98fbba0fd03c 11 seconds ago 657 MB
然后可以将这个上传到私有仓库里面，方便其他Node节点下载，如下所示：
1 [root@k8s-master ~]# docker tag tomcat-book:latest 192.168.110.133:5000/tomcat-book:latest 2 [root@k8s-master ~]# docker push 192.168.110.133:5000/tomcat-book:latest
然后创建tomcat的rc、svc，如下所示：
1 [root@k8s-master tomcat_demo]# kubectl create -f tomcat-rc.yml 2 replicationcontroller "myweb" created 3 [root@k8s-master tomcat_demo]# kubectl get all -o wide 4 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 5 rc/mysql 1 1 1 2h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 6 rc/myweb 1 1 1 7s myweb 192.168.110.133:5000/tomcat-book:latest app=myweb 7 8 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 9 svc/kubernetes 10.254.0.1 <none> 443/TCP 24d <none> 10 svc/mysql 10.254.234.52 <none> 3306/TCP 2h app=mysql 11 12 NAME READY STATUS RESTARTS AGE IP NODE 13 po/mysql-4j7qk 1/1 Running 0 2h 172.16.16.3 k8s-node3 14 po/myweb-cvd16 1/1 Running 0 7s 172.16.16.5 k8s-node3 15 [root@k8s-master tomcat_demo]# kubectl describe pod myweb-cvd16 16 Name: myweb-cvd16 17 Namespace: default 18 Node: k8s-node3/192.168.110.135 19 Start Time: Mon, 29 Jun 2020 20:12:54 +0800 20 Labels: app=myweb 21 Status: Running 22 IP: 172.16.16.5 23 Controllers: ReplicationController/myweb 24 Containers: 25 myweb: 26 Container ID: docker://1caf37bb344e394af5701035f9a28631129260f4cb5a5f1f88a214a66891631e 27 Image: 192.168.110.133:5000/tomcat-book:latest 28 Image ID: docker-pullable://192.168.110.133:5000/tomcat-book@sha256:e838afdd7d378bd078dbf431d5f956a197b94caba624b238049108e64cf66526 29 Port: 8080/TCP 30 State: Running 31 Started: Mon, 29 Jun 2020 20:12:55 +0800 32 Ready: True 33 Restart Count: 0 34 Volume Mounts: <none> 35 Environment Variables: 36 MYSQL_SERVICE_HOST: 10.254.234.52 37 MYSQL_SERVICE_PORT: 3306 38 Conditions: 39 Type Status 40 Initialized True 41 Ready True 42 PodScheduled True 43 No volumes. 44 QoS Class: BestEffort 45 Tolerations: <none> 46 Events: 47 FirstSeen LastSeen Count From SubObjectPath Type Reason Message 48 --------- -------- ----- ---- ------------- -------- ------ ------- 49 22s 22s 1 {default-scheduler } Normal Scheduled Successfully assigned myweb-cvd16 to k8s-node3 50 21s 21s 1 {kubelet k8s-node3} spec.containers{myweb} Normal Pulling pulling image "192.168.110.133:5000/tomcat-book:latest" 51 21s 21s 1 {kubelet k8s-node3} spec.containers{myweb} Normal Pulled Successfully pulled image "192.168.110.133:5000/tomcat-book:latest" 52 21s 21s 1 {kubelet k8s-node3} spec.containers{myweb} Normal Created Created container with docker id 1caf37bb344e; Security:[seccomp=unconfined] 53 21s 21s 1 {kubelet k8s-node3} spec.containers{myweb} Normal Started Started container with docker id 1caf37bb344e 54 [root@k8s-master tomcat_demo]# kubectl create -f tomcat-svc.yml 55 service "myweb" created 56 [root@k8s-master tomcat_demo]# kubectl get all -o wide 57 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 58 rc/mysql 1 1 1 2h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 59 rc/myweb 1 1 1 34s myweb 192.168.110.133:5000/tomcat-book:latest app=myweb 60 61 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 62 svc/kubernetes 10.254.0.1 <none> 443/TCP 24d <none> 63 svc/mysql 10.254.234.52 <none> 3306/TCP 2h app=mysql 64 svc/myweb 10.254.132.182 <nodes> 8080:30008/TCP 4s app=myweb 65 66 NAME READY STATUS RESTARTS AGE IP NODE 67 po/mysql-4j7qk 1/1 Running 0 2h 172.16.16.3 k8s-node3 68 po/myweb-cvd16 1/1 Running 0 34s 172.16.16.5 k8s-node3 69 [root@k8s-master tomcat_demo]#
访问地址http://192.168.110.133:30008/，报错了，源服务器未能找到目标资源的表示或者是不愿公开一个已经存在的资源表示。
还按照之前搞过的，这次找到是那台机器运行了这个Pod，找到了，然后将webapps.dist目录里面的文件拷贝到webapps里面就可以了。
1 [root@k8s-node3 ~]# docker ps 2 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3 1caf37bb344e 192.168.110.133:5000/tomcat-book:latest "catalina.sh run" 6 minutes ago Up 6 minutes k8s_myweb.2d2942f9_myweb-cvd16_default_dc959aad-ba01-11ea-9f79-000c2919d52d_c7802e7f 4 5683c0b919d2 192.168.110.133:5000/pod-infrastructure:latest "/pod" 6 minutes ago Up 6 minutes k8s_POD.37d60359_myweb-cvd16_default_dc959aad-ba01-11ea-9f79-000c2919d52d_919c83e8 5 9b536ac00a54 192.168.110.133:5000/mysql:5.7.30 "docker-entrypoint..." 2 hours ago Up 2 hours k8s_mysql.10e32324_mysql-4j7qk_default_0385e04b-b9ef-11ea-9f79-000c2919d52d_9c9a5b84 6 e4ff14740839 192.168.110.133:5000/pod-infrastructure:latest "/pod" 2 hours ago Up 2 hours k8s_POD.30310355_mysql-4j7qk_default_0385e04b-b9ef-11ea-9f79-000c2919d52d_3a453e5f 7 d2f4e384b94c 192.168.110.133:5000/kubernetes-dashboard-amd64:v1.5.0 "/dashboard --port..." 9 hours ago Up 9 hours k8s_kubernetes-dashboard.4dfe9ff5_kubernetes-dashboard-latest-3333846798-3mm68_kube-system_8a34f648-b8f4-11ea-91a9-000c2919d52d_d5314b66 8 eac876e090b7 192.168.110.133:5000/pod-infrastructure:latest "/pod" 9 hours ago Up 9 hours k8s_POD.3ba4035b_kubernetes-dashboard-latest-3333846798-3mm68_kube-system_8a34f648-b8f4-11ea-91a9-000c2919d52d_c44ca4ec 9 77da48b3cd89 registry "/entrypoint.sh /e..." 11 days ago Up 9 hours 0.0.0.0:5000->5000/tcp registry 10 [root@k8s-node3 ~]# docker exec -it 1caf37bb344e /bin/bash 11 root@myweb-cvd16:/usr/local/tomcat# ls 12 BUILDING.txt CONTRIBUTING.md LICENSE NOTICE README.md RELEASE-NOTES RUNNING.txt bin conf lib logs native-jni-lib temp webapps webapps.dist work 13 root@myweb-cvd16:/usr/local/tomcat# cd webapps 14 root@myweb-cvd16:/usr/local/tomcat/webapps# ls 15 book-master book-master.war 16 root@myweb-cvd16:/usr/local/tomcat/webapps# cd .. 17 root@myweb-cvd16:/usr/local/tomcat# ls 18 BUILDING.txt CONTRIBUTING.md LICENSE NOTICE README.md RELEASE-NOTES RUNNING.txt bin conf lib logs native-jni-lib temp webapps webapps.dist work 19 root@myweb-cvd16:/usr/local/tomcat# cd webapps.dist/ 20 root@myweb-cvd16:/usr/local/tomcat/webapps.dist# ls 21 ROOT docs examples host-manager manager 22 root@myweb-cvd16:/usr/local/tomcat/webapps.dist# cp -r * ../webapps 23 root@myweb-cvd16:/usr/local/tomcat/webapps.dist# ls 24 ROOT docs examples host-manager manager 25 root@myweb-cvd16:/usr/local/tomcat/webapps.dist# cd .. 26 root@myweb-cvd16:/usr/local/tomcat# cd webapps 27 root@myweb-cvd16:/usr/local/tomcat/webapps# ls 28 ROOT book-master book-master.war docs examples host-manager manager 29 root@myweb-cvd16:/usr/local/tomcat/webapps#
访问地址http://192.168.110.133:30008/，如下所示：
访问地址http://192.168.110.133:30008/book-master/，如下所示：
需要注意的是，我的项目连接的mysql的地址就是Master节点的ip地址的，然后Pod服务与服务之间的访问，比如tomcat访问mysql的是tomcat连的是mysql的svc的地址，如下所示：
1 drivername=com.mysql.jdbc.Driver 2 url=jdbc:mysql://192.168.110.133:3306/book?useUnicode=true&characterEncoding=utf8 3 user=root 4 password=123456
运行项目可以正常运行，简单的效果，如下所示：
查看kubernetes dashboard的界面如下所示：
点击可以查看详细的日志信息，如下所示：
可以通过命令对Pod的副本数进行调整，如下所示：
1 [root@k8s-master ~]# kubectl scale rc myweb --replicas=2 2 replicationcontroller "myweb" scaled 3 [root@k8s-master ~]# kubectl get all -o wide 4 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 5 rc/mysql 1 1 1 3h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 6 rc/myweb 2 2 1 59m myweb 192.168.110.133:5000/tomcat-book:latest app=myweb 7 8 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 9 svc/kubernetes 10.254.0.1 <none> 443/TCP 24d <none> 10 svc/mysql 10.254.234.52 <none> 3306/TCP 3h app=mysql 11 svc/myweb 10.254.132.182 <nodes> 8080:30008/TCP 59m app=myweb 12 13 NAME READY STATUS RESTARTS AGE IP NODE 14 po/mysql-4j7qk 1/1 Running 0 3h 172.16.16.3 k8s-node3 15 po/myweb-cvd16 1/1 Running 0 59m 172.16.16.5 k8s-node3 16 po/myweb-sk8qp 0/1 ContainerCreating 0 2s <none> k8s-node2 17 [root@k8s-master ~]# kubectl get all -o wide 18 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 19 rc/mysql 1 1 1 3h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 20 rc/myweb 2 2 2 1h myweb 192.168.110.133:5000/tomcat-book:latest app=myweb 21 22 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 23 svc/kubernetes 10.254.0.1 <none> 443/TCP 24d <none> 24 svc/mysql 10.254.234.52 <none> 3306/TCP 3h app=mysql 25 svc/myweb 10.254.132.182 <nodes> 8080:30008/TCP 59m app=myweb 26 27 NAME READY STATUS RESTARTS AGE IP NODE 28 po/mysql-4j7qk 1/1 Running 0 3h 172.16.16.3 k8s-node3 29 po/myweb-cvd16 1/1 Running 0 1h 172.16.16.5 k8s-node3 30 po/myweb-sk8qp 1/1 Running 0 59s 172.16.94.5 k8s-node2 31 [root@k8s-master ~]#
创建的时间大概几十秒吧，我的私有仓库包含项目的tomcat有七八百兆左右，也可以通过kubernetes dashboard进行查看，如下所示：
后来测试如何保证删除了mysql的pod如何保证数据不丢失的时候，发现一个问题的，我发现不管我怎么删除mysql的Pod之后，rc会立马启动一个mysql，但是我的数据就是没有丢失呢，我此时还没有做PV、PVC呢，后来发现原来我用的是192.168.110.133机器安装的mysql，也是服气子了，所以重新改了配置文件，然后重新创建mysql、myweb。下面开始创建mysql，如下所示：
1 [root@k8s-master ~]# cd k8s/ 2 [root@k8s-master k8s]# ls 3 book-master.war dashboard dashboard.zip deploy health heapster hpa metrics namespace pod rc skydns skydns.zip svc tomcat_demo tomcat_demo.zip volume 4 [root@k8s-master k8s]# cd tomcat_demo/ 5 [root@k8s-master tomcat_demo]# ls 6 book-master2.0.war book-master3.0.war book-master.war Dockerfile mysql-rc.yml mysql-svc.yml tomcat-rc.yml tomcat-svc.yml 7 [root@k8s-master tomcat_demo]# kubectl create -f mysql-rc.yml 8 replicationcontroller "mysql" created 9 [root@k8s-master tomcat_demo]# kubectl create -f mysql-svc.yml 10 service "mysql" created 11 [root@k8s-master tomcat_demo]# kubectl get all -o wide 12 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 13 rc/mysql 1 1 1 22s mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 14 15 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 16 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none> 17 svc/mysql 10.254.133.121 <none> 3306/TCP 17s app=mysql 18 19 NAME READY STATUS RESTARTS AGE IP NODE 20 po/mysql-2jtr6 1/1 Running 0 22s 172.16.16.3 k8s-node3 21 [root@k8s-master tomcat_demo]#
创建完毕Mysql之后，如果通过k8s进入到mysql呢，如下所示：
1 [root@k8s-master tomcat_demo]# kubectl exec -it mysql-2jtr6 bash 2 root@mysql-2jtr6:/# mysql -uroot -p123456 3 mysql: [Warning] Using a password on the command line interface can be insecure. 4 Welcome to the MySQL monitor. Commands end with ; or g. 5 Your MySQL connection id is 2 6 Server version: 5.7.30 MySQL Community Server (GPL) 7 8 Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. 9 10 Oracle is a registered trademark of Oracle Corporation and/or its 11 affiliates. Other names may be trademarks of their respective 12 owners. 13 14 Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. 15 16 mysql> show databases; 17 +--------------------+ 18 | Database | 19 +--------------------+ 20 | information_schema | 21 | mysql | 22 | performance_schema | 23 | sys | 24 +--------------------+ 25 4 rows in set (0.00 sec) 26 27 mysql> CREATE database book DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci; 28 Query OK, 1 row affected (0.00 sec) 29 30 mysql>
同样，也可以通过访问Pod的ip地址来访问Mysql服务的，如下所示：
1 [root@k8s-master tomcat_demo]# kubectl get all -o wide 2 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 3 rc/mysql 1 1 1 12h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 4 rc/myweb 1 1 1 11h myweb 192.168.110.133:5000/tomcat-book:4.0 app=myweb 5 6 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 7 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none> 8 svc/mysql 10.254.133.121 <none> 3306/TCP 12h app=mysql 9 svc/myweb 10.254.45.162 <nodes> 8080:30008/TCP 11h app=myweb 10 11 NAME READY STATUS RESTARTS AGE IP NODE 12 po/mysql-2jtr6 1/1 Running 0 12h 172.16.16.3 k8s-node3 13 po/myweb-9qhk9 1/1 Running 0 11h 172.16.16.5 k8s-node3 14 [root@k8s-master tomcat_demo]# mysql -uroot -p123456 -h172.16.16.3 15 mysql: [Warning] Using a password on the command line interface can be insecure. 16 Welcome to the MySQL monitor. Commands end with ; or g. 17 Your MySQL connection id is 29 18 Server version: 5.7.30 MySQL Community Server (GPL) 19 20 Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. 21 22 Oracle is a registered trademark of Oracle Corporation and/or its 23 affiliates. Other names may be trademarks of their respective 24 owners. 25 26 Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. 27 28 mysql>
然后可以创建自己的数据表，由于执行创建数据表的sql过长，这里进行了省略哦，如下所示：
1 mysql> show databases; 2 +--------------------+ 3 | Database | 4 +--------------------+ 5 | information_schema | 6 | book | 7 | mysql | 8 | performance_schema | 9 | sys | 10 +--------------------+ 11 5 rows in set (0.00 sec) 12 13 mysql> use book; 14 Database changed 15 mysql> show tables; 16 Empty set (0.00 sec) 17 18 mysql>
创建好数据表进行查看，如下所示：
1 mysql> show tables; 2 +---------------------+ 3 | Tables_in_book | 4 +---------------------+ 5 | book_info | 6 | book_recommendation | 7 | book_sort | 8 | user_book | 9 | user_info | 10 +---------------------+ 11 5 rows in set (0.00 sec) 12 13 mysql>
然后创建已经将2.x版本的项目打到tomcat的rc、svc，如下所示：
注意：其实这里为了搞清楚java项目的mysql的ip地址、端口号这些填啥，我已经打了很多版本了，最新的是4.0版本的了，操作几乎一样，但是注意下面的配置。
1 [root@k8s-master tomcat_demo]# kubectl create -f tomcat-rc.yml 2 replicationcontroller "myweb" created 3 [root@k8s-master tomcat_demo]# kubectl create -f tomcat-svc.yml 4 service "myweb" created 5 [root@k8s-master tomcat_demo]# kubectl get all -o wide 6 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 7 rc/mysql 1 1 1 6m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 8 rc/myweb 1 1 1 12s myweb 192.168.110.133:5000/tomcat-book:2.0 app=myweb 9 10 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 11 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none> 12 svc/mysql 10.254.133.121 <none> 3306/TCP 6m app=mysql 13 svc/myweb 10.254.117.144 <nodes> 8080:30008/TCP 8s app=myweb 14 15 NAME READY STATUS RESTARTS AGE IP NODE 16 po/mysql-2jtr6 1/1 Running 0 6m 172.16.16.3 k8s-node3 17 po/myweb-965n1 1/1 Running 0 12s 172.16.16.5 k8s-node3 18 [root@k8s-master tomcat_demo]#
然后开始访问地址http://192.168.110.133:30008/book-master2.0/，k8s访问tomcat报错，源服务器未能找到目标资源的表示或者是不愿公开一个已经存在的资源表示。还可以按照上面的方法进行解决，这里再贴下步骤了，如下所示:
1 [root@k8s-node3 ~]# docker ps 2 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3 a242e75ee4fa 192.168.110.133:5000/tomcat-book:2.0 "catalina.sh run" 3 minutes ago Up 3 minutes k8s_myweb.fc01434c_myweb-965n1_default_b63cbb65-ba10-11ea-9f79-000c2919d52d_dec12b6a 4 e67764bb9b4b 192.168.110.133:5000/pod-infrastructure:latest "/pod" 3 minutes ago Up 3 minutes k8s_POD.37d60359_myweb-965n1_default_b63cbb65-ba10-11ea-9f79-000c2919d52d_57e633de 5 3220e5f8475c 192.168.110.133:5000/mysql:5.7.30 "docker-entrypoint..." 10 minutes ago Up 10 minutes k8s_mysql.10e32324_mysql-2jtr6_default_cbcf1d80-ba0f-11ea-9f79-000c2919d52d_99438f65 6 0d4e3411d100 192.168.110.133:5000/pod-infrastructure:latest "/pod" 10 minutes ago Up 10 minutes k8s_POD.30310355_mysql-2jtr6_default_cbcf1d80-ba0f-11ea-9f79-000c2919d52d_9504bf1f 7 d2f4e384b94c 192.168.110.133:5000/kubernetes-dashboard-amd64:v1.5.0 "/dashboard --port..." 11 hours ago Up 11 hours k8s_kubernetes-dashboard.4dfe9ff5_kubernetes-dashboard-latest-3333846798-3mm68_kube-system_8a34f648-b8f4-11ea-91a9-000c2919d52d_d5314b66 8 eac876e090b7 192.168.110.133:5000/pod-infrastructure:latest "/pod" 11 hours ago Up 11 hours k8s_POD.3ba4035b_kubernetes-dashboard-latest-3333846798-3mm68_kube-system_8a34f648-b8f4-11ea-91a9-000c2919d52d_c44ca4ec 9 77da48b3cd89 registry "/entrypoint.sh /e..." 12 days ago Up 11 hours 0.0.0.0:5000->5000/tcp registry 10 [root@k8s-node3 ~]# docker exec -it a242e75ee4fa /bin/bash 11 root@myweb-965n1:/usr/local/tomcat# cd webapps.dist/ 12 root@myweb-965n1:/usr/local/tomcat/webapps.dist# cp -r * ../webapps 13 root@myweb-965n1:/usr/local/tomcat/webapps.dist# ls 14 ROOT docs examples host-manager manager 15 root@myweb-965n1:/usr/local/tomcat/webapps.dist# cd ../webapps 16 root@myweb-965n1:/usr/local/tomcat/webapps#
注意，我这里的java项目使用的就是svc/mysql的ip地址10.254.133.12，然后tomcat-rc.yml由于配置了dns，直接使用的是mysql这个svc的名称。
1 [root@k8s-master tomcat_demo]# kubectl get all -o wide 2 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 3 rc/mysql 1 1 1 48m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 4 rc/myweb 1 1 1 5m myweb 192.168.110.133:5000/tomcat-book:4.0 app=myweb 5 6 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 7 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none> 8 svc/mysql 10.254.133.121 <none> 3306/TCP 48m app=mysql 9 svc/myweb 10.254.45.162 <nodes> 8080:30008/TCP 5m app=myweb 10 11 NAME READY STATUS RESTARTS AGE IP NODE 12 po/mysql-2jtr6 1/1 Running 0 48m 172.16.16.3 k8s-node3 13 po/myweb-9qhk9 1/1 Running 0 5m 172.16.16.5 k8s-node3 14 [root@k8s-master tomcat_demo]#
注意tomcat-rc.yml里面的mysql的ip地址直接使用了mysql的svc名称，如果没有配置dns就需要每次修改mysql的svc的ip地址。
1 [root@k8s-master tomcat_demo]# cat tomcat-rc.yml 2 apiVersion: v1 3 kind: ReplicationController 4 metadata: 5 name: myweb 6 spec: 7 replicas: 1 8 selector: 9 app: myweb 10 template: 11 metadata: 12 labels: 13 app: myweb 14 spec: 15 containers: 16 - name: myweb 17 # image: 192.168.110.133:5000/tomcat:latest 18 image: 192.168.110.133:5000/tomcat-book:4.0 19 ports: 20 - containerPort: 8080 21 env: 22 - name: MYSQL_SERVICE_HOST 23 # value: '192.168.110.133' 24 # value: '10.254.234.52' 25 value: 'mysql' 26 - name: MYSQL_SERVICE_PORT 27 value: '3306' 28 [root@k8s-master tomcat_demo]#
我的java项目的连接mysql数据库的ip地址和端口号直接使用的mysql的svc的ip地址，端口号，这样的缺点就是每次创建mysql都会随机分配一个新的ip地址，就很麻烦的。
1 drivername=com.mysql.jdbc.Driver 2 url=jdbc:mysql://10.254.133.121:3306/book?useUnicode=true&characterEncoding=utf8 3 user=root 4 password=123456
如果按照上面的链接k8s创建的mysql，如果mysql的svc即service重新建一个，会导致这个java连不上mysql数据库的，我又使用了下面这个配置连接k8s里面的mysql数据库，这里使用了svc（service）的名称来代替，如果配置好了dns的话，是可以这样配置的，我的tomcat-rc.yml连接k8s创建的mysql就是这样配置的，如下所示：
1 drivername=com.mysql.jdbc.Driver 2 url=jdbc:mysql://mysql:3306/book?useUnicode=true&characterEncoding=utf8 3 user=root 4 password=123456
此时我的tomcat-rc.yml是这个样子的。
1 [root@k8s-master tomcat_demo]# cat tomcat-rc.yml 2 apiVersion: v1 3 kind: ReplicationController 4 metadata: 5 name: myweb 6 spec: 7 replicas: 1 8 selector: 9 app: myweb 10 template: 11 metadata: 12 labels: 13 app: myweb 14 spec: 15 containers: 16 - name: myweb 17 # image: 192.168.110.133:5000/tomcat:latest 18 image: 192.168.110.133:5000/tomcat-book:5.0 19 ports: 20 - containerPort: 8080 21 env: 22 - name: MYSQL_SERVICE_HOST 23 # value: '192.168.110.133' 24 # value: '10.254.234.52' 25 value: 'mysql' 26 - name: MYSQL_SERVICE_PORT 27 value: '3306' 28 [root@k8s-master tomcat_demo]#
此时，删除mysql的pod验证一下，自己java web如此配置连接k8s里面的Mysql是否正常，然后验证数据没有持久化会造成数据丢失吗？
1 [root@k8s-master tomcat_demo]# kubectl get all -o wide 2 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 3 rc/mysql 1 1 1 12h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 4 rc/myweb 1 1 1 9m myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb 5 6 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 7 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none> 8 svc/mysql 10.254.133.121 <none> 3306/TCP 12h app=mysql 9 svc/myweb 10.254.180.131 <nodes> 8080:30008/TCP 9m app=myweb 10 11 NAME READY STATUS RESTARTS AGE IP NODE 12 po/mysql-2jtr6 1/1 Running 0 12h 172.16.16.3 k8s-node3 13 po/myweb-k23p0 1/1 Running 0 9m 172.16.16.5 k8s-node3 14 [root@k8s-master tomcat_demo]# kubectl delete pod mysql-2jtr6 15 pod "mysql-2jtr6" deleted 16 [root@k8s-master tomcat_demo]# kubectl get all -o wide 17 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 18 rc/mysql 1 1 1 13h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 19 rc/myweb 1 1 1 15m myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb 20 21 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 22 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none> 23 svc/mysql 10.254.133.121 <none> 3306/TCP 13h app=mysql 24 svc/myweb 10.254.180.131 <nodes> 8080:30008/TCP 14m app=myweb 25 26 NAME READY STATUS RESTARTS AGE IP NODE 27 po/mysql-2jtr6 1/1 Terminating 0 13h 172.16.16.3 k8s-node3 28 po/mysql-kt2v2 1/1 Running 0 2s 172.16.16.6 k8s-node3 29 po/myweb-k23p0 1/1 Running 0 15m 172.16.16.5 k8s-node3 30 [root@k8s-master tomcat_demo]#
最终会发现你删除的那个mysql的pod被删除之前，新的mysql的pod已经创建成功了的。
1 [root@k8s-master tomcat_demo]# kubectl get all -o wide 2 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 3 rc/mysql 1 1 1 13h mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 4 rc/myweb 1 1 1 16m myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb 5 6 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 7 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none> 8 svc/mysql 10.254.133.121 <none> 3306/TCP 13h app=mysql 9 svc/myweb 10.254.180.131 <nodes> 8080:30008/TCP 15m app=myweb 10 11 NAME READY STATUS RESTARTS AGE IP NODE 12 po/mysql-kt2v2 1/1 Running 0 1m 172.16.16.6 k8s-node3 13 po/myweb-k23p0 1/1 Running 0 16m 172.16.16.5 k8s-node3 14 [root@k8s-master tomcat_demo]#
然后登录mysql，这里通过mysql的pod的ip地址进行登录，发现数据已经丢失了，如下所示：
1 [root@k8s-master tomcat_demo]# mysql -uroot -p123456 -h172.16.16.6 2 mysql: [Warning] Using a password on the command line interface can be insecure. 3 Welcome to the MySQL monitor. Commands end with ; or g. 4 Your MySQL connection id is 4 5 Server version: 5.7.30 MySQL Community Server (GPL) 6 7 Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. 8 9 Oracle is a registered trademark of Oracle Corporation and/or its 10 affiliates. Other names may be trademarks of their respective 11 owners. 12 13 Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. 14 15 mysql> show databases; 16 +--------------------+ 17 | Database | 18 +--------------------+ 19 | information_schema | 20 | mysql | 21 | performance_schema | 22 | sys | 23 +--------------------+ 24 4 rows in set (0.01 sec) 25 26 mysql>
或者可以使用k8s的命令进入到Mysql，进行登录，如下所示：
1 [root@k8s-master tomcat_demo]# kubectl exec -it mysql-kt2v2 bash 2 root@mysql-kt2v2:/# mysql -uroot -p123456 3 mysql: [Warning] Using a password on the command line interface can be insecure. 4 Welcome to the MySQL monitor. Commands end with ; or g. 5 Your MySQL connection id is 5 6 Server version: 5.7.30 MySQL Community Server (GPL) 7 8 Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. 9 10 Oracle is a registered trademark of Oracle Corporation and/or its 11 affiliates. Other names may be trademarks of their respective 12 owners. 13 14 Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. 15 16 mysql> show databases; 17 +--------------------+ 18 | Database | 19 +--------------------+ 20 | information_schema | 21 | mysql | 22 | performance_schema | 23 | sys | 24 +--------------------+ 25 4 rows in set (0.00 sec) 26 27 mysql>
注意：如果Mysql的mysql-svc.yml，这个svc配置了NodePort方式，对外暴漏了端口号，那么在服务器可以通过访问暴漏的端口号，和服务器的ip地址，已经mysql的账号，密码就可以访问k8s里面创建的mysql了的。总之，访问方式有很多种，自己理清楚，自己是通过pod的ip地址（mysql -uroot -p123456 -h172.16.16.6），还是通过k8s的命令访问mysql（kubectl exec -it mysql-kt2v2 bash，然后使用mysql -uroot -p123456查看数据库信息），还是通过svc对外暴漏端口号的方式（这种方式需要配置mysql的svc的yaml配置文件，我这里没有对外暴漏端口号，自己需要的话，可以对外进行暴漏的）。
5、如果上面的Java Web项目没有做持久化之前，如果mysql的pod删除掉，由于配置了rc，k8s会里面启动一个mysql，但是会发现新增的数据没有了，那么如何使用PV、PVC保证数据不丢失呢？这里只是做了mysql的持久化操作。
1 [root@k8s-master ~]# cd k8s/ 2 [root@k8s-master k8s]# ls 3 book-master.war dashboard dashboard.zip deploy health heapster hpa metrics namespace pod rc skydns skydns.zip svc tomcat_demo tomcat_demo.zip volume 4 [root@k8s-master k8s]# cd tomcat_demo/ 5 [root@k8s-master tomcat_demo]# ls 6 book-master2.0.war book-master3.0.war book-master4.0.war book-master5.0.war book-master6.0.war book-master.war Dockerfile mysql-rc.yml mysql-svc.yml tomcat-rc.yml tomcat-svc.yml 7 [root@k8s-master tomcat_demo]# cp ../volume/* . 8 [root@k8s-master tomcat_demo]# ls 9 book-master2.0.war book-master4.0.war book-master6.0.war Dockerfile mysql-svc.yml test-pv.yaml tomcat-svc.yml 10 book-master3.0.war book-master5.0.war book-master.war mysql-rc.yml test-pvc.yaml tomcat-rc.yml 11 [root@k8s-master tomcat_demo]# mv test-pv.yaml mysql-pv.yaml 12 [root@k8s-master tomcat_demo]# mv test-pvc.yaml mysql-pvc.yaml 13 [root@k8s-master tomcat_demo]# ls 14 book-master2.0.war book-master4.0.war book-master6.0.war Dockerfile mysql-pv.yaml mysql-svc.yml tomcat-svc.yml 15 book-master3.0.war book-master5.0.war book-master.war mysql-pvc.yaml mysql-rc.yml tomcat-rc.yml 16 [root@k8s-master tomcat_demo]#
编辑mysql-pv.yaml、mysql-pvc.yaml两个配置文件，如下所示：
1 apiVersion: v1 2 kind: PersistentVolume 3 metadata: 4 name: mysql 5 labels: 6 type: mysql 7 spec: 8 capacity: 9 storage: 5Gi # 实际生产环境，可以搞大一点的。 10 accessModes: 11 - ReadWriteMany 12 persistentVolumeReclaimPolicy: Recycle 13 nfs: 14 path: "/data/mysql" #拿出一个目录给Mysql来使用。 15 server: 192.168.110.133 16 readOnly: false
将全部的test替换为mysql的用法，如下所示：
然后创建/data/mysql的目录，如下所示：
1 [root@k8s-master tomcat_demo]# mkdir /data/mysql 2 [root@k8s-master tomcat_demo]# ls /data/mysql/
然后修改mysql-pvc.yaml配置文件，如下所示：
1 kind: PersistentVolumeClaim 2 apiVersion: v1 3 metadata: 4 name: mysql 5 spec: 6 accessModes: 7 - ReadWriteMany 8 resources: 9 requests: 10 storage: 5Gi
然后依次创建mysq的PV（全局的资源，被绑定到了default/mysqlpvc这个pvc了）、PVC，并进行查看，如下所示：
1 [root@k8s-master tomcat_demo]# ls 2 book-master2.0.war book-master4.0.war book-master6.0.war Dockerfile mysql-pv.yaml mysql-rc.yml tomcat-rc.yml 3 book-master3.0.war book-master5.0.war book-master.war mysql-pvc.yaml mysql-rc-pvc.yml mysql-svc.yml tomcat-svc.yml 4 [root@k8s-master tomcat_demo]# kubectl create -f mysql-pv.yaml 5 persistentvolume "mysql" created 6 [root@k8s-master tomcat_demo]# kubectl create -f mysql-pvc.yaml 7 persistentvolumeclaim "mysql" created 8 [root@k8s-master tomcat_demo]# kubectl get pvc 9 NAME STATUS VOLUME CAPACITY ACCESSMODES AGE 10 mysql Bound mysql 5Gi RWX 6s 11 [root@k8s-master tomcat_demo]# kubectl get pv 12 NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE 13 mysql 5Gi RWX Recycle Bound default/mysql 15s 14 [root@k8s-master tomcat_demo]#
那么，现在配置mysql的rc，让它使用卷pvc，这样就可以完成了数据持久化存储了。
1 [root@k8s-master tomcat_demo]# cp mysql-rc.yml mysql-rc-pvc.yml 2 [root@k8s-master tomcat_demo]# vim mysql-rc-pvc.yml
使用的方式就是先增加一个目录，将这个目录挂载出来，比如将/var/lib/mysql目录挂载出来，Pod里面可以有多个容器，所以卷的名字有时候需要写多个，写完多个之后，在使用卷的时候再来声明多个，即volumes的名字必须和volumeMounts挂载的名字是一样的。
具体内容，如下所示：
1 apiVersion: v1 2 kind: ReplicationController 3 metadata: 4 name: mysql 5 spec: 6 replicas: 1 7 selector: 8 app: mysql 9 template: 10 metadata: 11 labels: 12 app: mysql 13 spec: 14 containers: 15 - name: mysql 16 image: 192.168.110.133:5000/mysql:5.7.30 17 ports: 18 - containerPort: 3306 19 env: 20 - name: MYSQL_ROOT_PASSWORD 21 value: '123456' 22 volumeMounts: 23 - name: data 24 mountPath: /var/lib/mysql 25 volumes: 26 - name: data 27 persistentVolumeClaim: 28 claimName: mysql
此时，将mysql-rc-pvc.yml应用一下子，如下所示：
1 [root@k8s-master tomcat_demo]# kubectl apply -f mysql-rc-pvc.yml 2 replicationcontroller "mysql" configured 3 [root@k8s-master tomcat_demo]# kubectl get all -o wide 4 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 5 rc/mysql 1 1 1 35m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 6 rc/myweb 1 1 1 35m myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb 7 8 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 9 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none> 10 svc/mysql 10.254.182.125 <none> 3306/TCP 35m app=mysql 11 svc/myweb 10.254.188.155 <nodes> 8080:30008/TCP 35m app=myweb 12 13 NAME READY STATUS RESTARTS AGE IP NODE 14 po/mysql-nt1zl 1/1 Running 1 35m 172.16.66.3 k8s-node3 15 po/myweb-c8sf6 1/1 Running 1 35m 172.16.66.5 k8s-node3 16 [root@k8s-master tomcat_demo]#
此时，虽然应用成功了，但是并没有启动一个新的mysql的pod，此时，需要将mysql的pod删除掉，让rc再创建一个新的mysql的pod，如下所示：
1 [root@k8s-master tomcat_demo]# kubectl delete pod mysql-nt1zl
嗯，很显然，我的这里出现问题了，删除之后，新的mysql的pod启动不起来了，然后我把mysql的rc、svc都删除了，启动新的mysql的rc、svc还是启动不起来了，这里查看一下具体错误信息，如下所示：
1 [root@k8s-master tomcat_demo]# kubectl describe pod mysql-p3mcj 2 Name: mysql-p3mcj 3 Namespace: default 4 Node: k8s-node3/192.168.110.135 5 Start Time: Tue, 30 Jun 2020 12:32:35 +0800 6 Labels: app=mysql 7 Status: Pending 8 IP: 9 Controllers: ReplicationController/mysql 10 Containers: 11 mysql: 12 Container ID: 13 Image: 192.168.110.133:5000/mysql:5.7.30 14 Image ID: 15 Port: 3306/TCP 16 State: Waiting 17 Reason: ContainerCreating 18 Ready: False 19 Restart Count: 0 20 Volume Mounts: 21 /var/lib/mysql from data (rw) 22 Environment Variables: 23 MYSQL_ROOT_PASSWORD: 123456 24 Conditions: 25 Type Status 26 Initialized True 27 Ready False 28 PodScheduled True 29 Volumes: 30 data: 31 Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) 32 ClaimName: mysql 33 ReadOnly: false 34 QoS Class: BestEffort 35 Tolerations: <none> 36 Events: 37 FirstSeen LastSeen Count From SubObjectPath Type Reason Message 38 --------- -------- ----- ---- ------------- -------- ------ ------- 39 2m 2m 1 {default-scheduler } Normal Scheduled Successfully assigned mysql-p3mcj to k8s-node3 40 53s 53s 1 {kubelet k8s-node3} Warning FailedMount Unable to mount volumes for pod "mysql-p3mcj_default(b949a085-ba8a-11ea-af28-000c2919d52d)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-p3mcj". list of unattached/unmounted volumes=[data] 41 53s 53s 1 {kubelet k8s-node3} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-p3mcj". list of unattached/unmounted volumes=[data] 42 47s 47s 1 {kubelet k8s-node3} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/nfs/b949a085-ba8a-11ea-af28-000c2919d52d-mysql" (spec.Name: "mysql") pod "b949a085-ba8a-11ea-af28-000c2919d52d" (UID: "b949a085-ba8a-11ea-af28-000c2919d52d") with: mount failed: exit status 32 43 Mounting command: mount 44 Mounting arguments: 192.168.110.133:/data/mysql /var/lib/kubelet/pods/b949a085-ba8a-11ea-af28-000c2919d52d/volumes/kubernetes.io~nfs/mysql nfs [] 45 Output: mount.nfs: Connection refused 46 47 48 [root@k8s-master tomcat_demo]#
出现这种挂载不了的问题，就是nfs没有配置好，下面可以再配置一下，如下所示：
关于多个centos7操作系统间实现文件数据资源共享（挂载mount网络文件路径），这边通过在服务端配置nfs（网络文件系统）+rpcbind服务软件，在三台机器上面确定安装了nfs，如下所示：
1 [root@k8s-master ~]# rpm -qa|grep nfs 2 nfs-utils-1.3.0-0.66.el7.x86_64 3 libnfsidmap-0.25-19.el7.x86_64 4 [root@k8s-master ~]#
确认三台服务端都安装配置rpcbind、如下所示：
1 [root@k8s-master ~]# service nfs start 2 Redirecting to /bin/systemctl start nfs.service
在服务端的/etc/exports配置共享路径，并重启服务，同时关闭防火墙（比较重要，否则报mount.nfs: Connection timed out）。
1 [root@k8s-master ~]# cat /etc/exports 2 # 共享data目录，允许192.168.110.*访问，即允许110段ip地址访问。读写权限，同步，不做root用户、所有用户的UID映射 3 /data 192.168.110.0/24(rw,async,no_root_squash,no_all_squash) 4 5 6 [root@k8s-master ~]#
设置/data目录的权限，如下所示：
1 [root@k8s-master ~]# ll /data/ 2 total 0 3 drwxr-xr-x 2 root root 6 Jun 29 17:21 k8s 4 drwxr-xr-x 5 polkitd root 328 Jun 30 13:04 mysql 5 [root@k8s-master ~]# chmod -R 755 /data/ 6 [root@k8s-master ~]# ll /data/ 7 total 0 8 drwxr-xr-x 2 root root 6 Jun 29 17:21 k8s 9 drwxr-xr-x 5 polkitd root 328 Jun 30 13:04 mysql 10 [root@k8s-master ~]#
在客户端mount上面服务端分享的nfs路径/data(绑定在客户端的/data)，测试成功。
1 [root@k8s-node2 ~]# mount -t nfs 192.168.110.133:/data /data 2 mount.nfs: mount point /data does not exist 3 [root@k8s-node2 ~]# cd mkdir /data 4 -bash: cd: mkdir: No such file or directory 5 [root@k8s-node2 ~]# mkdir /data 6 [root@k8s-node2 ~]# mount -t nfs 192.168.110.133:/data /data 7 [root@k8s-node2 ~]# cd /data/ 8 [root@k8s-node2 data]# ls 9 k8s mysql 10 [root@k8s-node2 data]#
挂载成功之后，重启两个客户端的nfs服务、rpcbind服务，如下所示：
1 [root@k8s-node2 data]# systemctl restart rpcbind.service 2 [root@k8s-node2 data]# systemctl restart nfs.service 3 [root@k8s-node2 data]# 4 5 [root@k8s-node3 data]# systemctl restart rpcbind.service 6 [root@k8s-node3 data]# systemctl restart nfs.service 7 [root@k8s-node3 data]#
此时，再次查看创建的mysql的Pod是否已经创建成功了，如下所示：
1 [root@k8s-master tomcat_demo]# kubectl describe pod mysql-njrhc 2 Name: mysql-njrhc 3 Namespace: default 4 Node: k8s-node3/192.168.110.135 5 Start Time: Tue, 30 Jun 2020 12:42:45 +0800 6 Labels: app=mysql 7 Status: Running 8 IP: 172.16.66.3 9 Controllers: ReplicationController/mysql 10 Containers: 11 mysql: 12 Container ID: docker://a689aeadd94774508b8f384befb62124e28d208801dda204aa4e36e24c476960 13 Image: 192.168.110.133:5000/mysql:5.7.30 14 Image ID: docker-pullable://192.168.110.133:5000/mysql@sha256:0563b36ec2d1a262f79e1d8562e61f642a0f64f93306d8a709047cdea0444d0a 15 Port: 3306/TCP 16 State: Running 17 Started: Tue, 30 Jun 2020 13:03:43 +0800 18 Ready: True 19 Restart Count: 0 20 Volume Mounts: 21 /var/lib/mysql from data (rw) 22 Environment Variables: 23 MYSQL_ROOT_PASSWORD: 123456 24 Conditions: 25 Type Status 26 Initialized True 27 Ready True 28 PodScheduled True 29 Volumes: 30 data: 31 Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) 32 ClaimName: mysql 33 ReadOnly: false 34 QoS Class: BestEffort 35 Tolerations: <none> 36 Events: 37 FirstSeen LastSeen Count From SubObjectPath Type Reason Message 38 --------- -------- ----- ---- ------------- -------- ------ ------- 39 27m 27m 1 {default-scheduler } Normal Scheduled Successfully assigned mysql-njrhc to k8s-node3 40 25m 9m 8 {kubelet k8s-node3} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/nfs/25204e19-ba8c-11ea-af28-000c2919d52d-mysql" (spec.Name: "mysql") pod "25204e19-ba8c-11ea-af28-000c2919d52d" (UID: "25204e19-ba8c-11ea-af28-000c2919d52d") with: mount failed: exit status 32 41 Mounting command: mount 42 Mounting arguments: 192.168.110.133:/data/mysql /var/lib/kubelet/pods/25204e19-ba8c-11ea-af28-000c2919d52d/volumes/kubernetes.io~nfs/mysql nfs [] 43 Output: mount.nfs: Connection refused 44 45 46 25m 7m 9 {kubelet k8s-node3} Warning FailedMount Unable to mount volumes for pod "mysql-njrhc_default(25204e19-ba8c-11ea-af28-000c2919d52d)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-njrhc". list of unattached/unmounted volumes=[data] 47 25m 7m 9 {kubelet k8s-node3} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-njrhc". list of unattached/unmounted volumes=[data] 48 6m 6m 1 {kubelet k8s-node3} spec.containers{mysql} Normal Pulled Container image "192.168.110.133:5000/mysql:5.7.30" already present on machine 49 6m 6m 1 {kubelet k8s-node3} spec.containers{mysql} Normal Created Created container with docker id a689aeadd947; Security:[seccomp=unconfined] 50 6m 6m 1 {kubelet k8s-node3} spec.containers{mysql} Normal Started Started container with docker id a689aeadd947 51 [root@k8s-master tomcat_demo]# kubectl get all -o wide 52 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 53 rc/mysql 1 1 1 38m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 54 rc/myweb 1 1 1 1h myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb 55 56 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 57 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none> 58 svc/mysql 10.254.126.11 <none> 3306/TCP 38m app=mysql 59 svc/myweb 10.254.188.155 <nodes> 8080:30008/TCP 1h app=myweb 60 61 NAME READY STATUS RESTARTS AGE IP NODE 62 po/mysql-njrhc 1/1 Running 0 28m 172.16.66.3 k8s-node3 63 po/myweb-c8sf6 1/1 Running 1 1h 172.16.66.5 k8s-node3 64 [root@k8s-master tomcat_demo]#
由于此时，两个客户端都挂载了服务器端的目录，此时查看三台机器的/data目录，发现里面的内容完全一样，如下所示：
1 [root@k8s-master ~]# cd /data/ 2 [root@k8s-master data]# ls 3 k8s mysql 4 [root@k8s-master data]# cd mysql/ 5 [root@k8s-master mysql]# ls 6 auto.cnf ca.pem client-key.pem ibdata1 ib_logfile1 mysql private_key.pem server-cert.pem sys 7 ca-key.pem client-cert.pem ib_buffer_pool ib_logfile0 ibtmp1 performance_schema public_key.pem server-key.pem 8 [root@k8s-master mysql]#
可以查看挂载信息，查看一下mysql的pod是在那个机器上启动的，然后使用命令查看挂载信息。
1 [root@k8s-master tomcat_demo]# kubectl get all -o wide 2 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 3 rc/mysql 1 1 1 46m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 4 rc/myweb 1 1 1 2h myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb 5 6 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 7 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none> 8 svc/mysql 10.254.126.11 <none> 3306/TCP 46m app=mysql 9 svc/myweb 10.254.188.155 <nodes> 8080:30008/TCP 2h app=myweb 10 11 NAME READY STATUS RESTARTS AGE IP NODE 12 po/mysql-njrhc 1/1 Running 0 36m 172.16.66.3 k8s-node3 13 po/myweb-c8sf6 1/1 Running 1 2h 172.16.66.5 k8s-node3 14 [root@k8s-master tomcat_demo]#
可以在Node3节点使用df -h命令查看挂载信息，如下所示：
1 [root@k8s-node3 mysql]# df -h 2 Filesystem Size Used Avail Use% Mounted on 3 /dev/mapper/centos-root 10G 6.5G 3.6G 65% / 4 devtmpfs 1.2G 0 1.2G 0% /dev 5 tmpfs 1.2G 0 1.2G 0% /dev/shm 6 tmpfs 1.2G 11M 1.2G 1% /run 7 tmpfs 1.2G 0 1.2G 0% /sys/fs/cgroup 8 /dev/sda1 197M 157M 41M 80% /boot 9 tmpfs 229M 12K 229M 1% /run/user/42 10 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/5265b3c878e49cfa04607df2b3d650eb4a66d2eff93edbabc9bbccffcb3cb5fb/merged 11 shm 64M 0 64M 0% /var/lib/docker/containers/77da48b3cd89ac31f2e1670978d6c4258bf5a5412b337a639776df4a60bc0264/shm 12 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/ab511d9e7a8f54e2ef11de6e809047bbfde443d1eb3244cbd0f6a8fe32d51552/merged 13 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/dc3decab78b1c5385384af3199727c41e7e05a96f769288cd133b1831e8b8cff/merged 14 shm 64M 0 64M 0% /var/lib/docker/containers/bad9be6adcd79953768767f69cab35515175fa08ac6cb924dadc40e8c52bd672/shm 15 shm 64M 0 64M 0% /var/lib/docker/containers/6700afcaaaa7115c9079a4ebb988dd6293edd6a5e880bf1b8d1bab4f023ae657/shm 16 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/2e1480e2559a602d0a0f5437113829ac5f50d2c09f38000aacdea87be0237616/merged 17 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/98519ce1d9e7d7712cc0aec0482fc0e3e246c9f8dfb9e6cbda8517e97b1a5063/merged 18 tmpfs 229M 0 229M 0% /run/user/0 19 192.168.110.133:/data/mysql 18G 14G 3.9G 79% /var/lib/kubelet/pods/25204e19-ba8c-11ea-af28-000c2919d52d/volumes/kubernetes.io~nfs/mysql 20 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/9cbf0877f0dc6abef9cfa3aba292066f36f7d7a12f50d642c98be91d62b97c1b/merged 21 shm 64M 0 64M 0% /var/lib/docker/containers/7e03fb4382a25120de8a8714a2c9210dd06b7bb09b94900f963e701d077b5843/shm 22 overlay 10G 6.5G 3.6G 65% /var/lib/docker/overlay2/64614fb76966f8d630d77bc333b24fa67c2078f711a05cf958b2e32c6bab36f5/merged 23 192.168.110.133:/data 18G 14G 3.9G 79% /data 24 [root@k8s-node3 mysql]#
然后可以在web界面录入几条新的数据，然后删除mysql的pod，开始验证是否已经做到持久化了，如下所示：
删除mysql的pod，操作如下所示：
1 [root@k8s-master tomcat_demo]# kubectl delete pod mysql-njrhc 2 pod "mysql-njrhc" deleted 3 [root@k8s-master tomcat_demo]# kubectl get all -o wide 4 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 5 rc/mysql 1 1 0 50m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 6 rc/myweb 1 1 1 2h myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb 7 8 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 9 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none> 10 svc/mysql 10.254.126.11 <none> 3306/TCP 50m app=mysql 11 svc/myweb 10.254.188.155 <nodes> 8080:30008/TCP 2h app=myweb 12 13 NAME READY STATUS RESTARTS AGE IP NODE 14 po/mysql-njrhc 1/1 Terminating 0 40m 172.16.66.3 k8s-node3 15 po/mysql-wldks 0/1 ContainerCreating 0 1s <none> k8s-node3 16 po/myweb-c8sf6 1/1 Running 1 2h 172.16.66.5 k8s-node3 17 [root@k8s-master tomcat_demo]# kubectl get all -o wide 18 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 19 rc/mysql 1 1 1 50m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 20 rc/myweb 1 1 1 2h myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb 21 22 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 23 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none> 24 svc/mysql 10.254.126.11 <none> 3306/TCP 50m app=mysql 25 svc/myweb 10.254.188.155 <nodes> 8080:30008/TCP 2h app=myweb 26 27 NAME READY STATUS RESTARTS AGE IP NODE 28 po/mysql-njrhc 1/1 Terminating 0 40m 172.16.66.3 k8s-node3 29 po/mysql-wldks 1/1 Running 0 3s 172.16.66.6 k8s-node3 30 po/myweb-c8sf6 1/1 Running 1 2h 172.16.66.5 k8s-node3 31 [root@k8s-master tomcat_demo]# kubectl get all -o wide 32 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 33 rc/mysql 1 1 1 50m mysql 192.168.110.133:5000/mysql:5.7.30 app=mysql 34 rc/myweb 1 1 1 2h myweb 192.168.110.133:5000/tomcat-book:5.0 app=myweb 35 36 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 37 svc/kubernetes 10.254.0.1 <none> 443/TCP 25d <none> 38 svc/mysql 10.254.126.11 <none> 3306/TCP 50m app=mysql 39 svc/myweb 10.254.188.155 <nodes> 8080:30008/TCP 2h app=myweb 40 41 NAME READY STATUS RESTARTS AGE IP NODE 42 po/mysql-wldks 1/1 Running 0 5s 172.16.66.6 k8s-node3 43 po/myweb-c8sf6 1/1 Running 1 2h 172.16.66.5 k8s-node3 44 [root@k8s-master tomcat_demo]#
然后查看界面上的新录入的数据，是否还存在，如下所示：
然后，查看一下挂载的目录也是有内容的，如下所示：
1 [root@k8s-node3 mysql]# ls 2 auto.cnf ca-key.pem client-cert.pem ib_buffer_pool ib_logfile0 ibtmp1 performance_schema public_key.pem server-key.pem 3 book ca.pem client-key.pem ibdata1 ib_logfile1 mysql private_key.pem server-cert.pem sys 4 [root@k8s-node3 mysql]#