kubernetes之滚动更新
滚动更新
滚动更新是一次只更新一小部分副本，成功后，在更新更多的副本，最终完成所有副本的更新，滚动更新的好处是零停机，整个过程始终有副本再运行，从而保证业务的连续性
下面我们不熟三副本应用，初始镜像为httpd：2.2 然后将其更新到httpd：2.4
httpd：2.2配置文件：
[root@master music]# cat httpd.yml apiVersion: apps/v1 kind: Deployment metadata: name: http-deploy labels: run: apache spec: replicas: 3 selector: matchLabels: run: apache template: metadata: labels: run: apache spec: containers: - name: httpd image: httpd:2.4 ports: - containerPort: 80
查看一下pod：
[root@master music]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES http-deploy-849cf97446-6k8jj 1/1 Running 0 2m28s 10.244.1.54 node1 <none> <none> http-deploy-849cf97446-l987p 1/1 Running 0 2m28s 10.244.1.55 node1 <none> <none> http-deploy-849cf97446-mtsqf 1/1 Running 0 2m28s 10.244.2.42 node2 <none> <none>
在查看一下当前版本：
[root@master music]# kubectl get replicasets.apps -o wide NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR http-deploy-849cf97446 3 3 3 10m httpd httpd:2.2 pod-template-hash=849cf97446,run=apache
现在我们来滚动更新： 把配置文件htppd.yml镜像httpd：2.2 更改为 httpd2.4，然后重新执行
现在我们再来看看
[root@master music]# kubectl get replicasets.apps -o wide NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR http-deploy-77c8788b9b 3 3 3 39s httpd httpd:2.4 pod-template-hash=77c8788b9b,run=apache http-deploy-849cf97446 0 0 0 13m httpd httpd:2.2 pod-template-hash=849cf97446,run=apache
发现了变化镜像2.2变成了2.4，重新创建了pod 镜像为2.4
[root@master music]# kubectl describe deployment Name: http-deploy Namespace: default CreationTimestamp: Mon, 20 Jul 2020 20:08:32 +0800 Labels: run=apache Annotations: deployment.kubernetes.io/revision: 2 kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"l Selector: run=apache Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: run=apache Containers: httpd: Image: httpd:2.4 Port: 80/TCP Host Port: 0/TCP Environment: <none> Mounts: <none> Volumes: <none> Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: <none> NewReplicaSet: http-deploy-77c8788b9b (3/3 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 17m deployment-controller Scaled up replica set http-deploy-849cf974 Normal ScalingReplicaSet 5m9s deployment-controller Scaled up replica set http-deploy-77c8788b Normal ScalingReplicaSet 4m52s deployment-controller Scaled down replica set http-deploy-849cf9 Normal ScalingReplicaSet 4m52s deployment-controller Scaled up replica set http-deploy-77c8788b Normal ScalingReplicaSet 4m35s deployment-controller Scaled down replica set http-deploy-849cf9 Normal ScalingReplicaSet 4m35s deployment-controller Scaled up replica set http-deploy-77c8788b Normal ScalingReplicaSet 4m34s deployment-controller Scaled down replica set http-deploy-849cf9
每次只更新替换一个pod，每次更换的pod数量是可以定制的，kubernetes提供了两个参数maxSurge和 maxUnavailable，来精细更换pod数量
回滚
kubectl apply 每次更新应用时 kubernetes都会记录下当然的配置，，保存为一个 revision（版次），这样就可以回滚到某个指定的revision
就是在执行的时候后面跟上一个参数， --record
下面我们来创建三个配置文件，三个文件版本不一样就可以我们用httpd：2.37，httpd：2.38，httpd：2.39
[root@master music]# cat httpd.yml apiVersion: apps/v1 kind: Deployment metadata: name: http-deploy labels: run: apache spec: replicas: 3 selector: matchLabels: run: apache template: metadata: labels: run: apache spec: containers: - name: httpd image: httpd:2.4.37 ##其余两个在这里就不写在这里了，把镜像版本改了就可以了 ports: - containerPort: 80
执行：
[root@master music]# kubectl apply -f httpd.yml --record deployment.apps/http-deploy created [root@master music]# kubectl apply -f httpd1.yml --record deployment.apps/http-deploy configured [root@master music]# kubectl apply -f httpd2.yml --record deployment.apps/http-deploy configured
通过查看可以看到每一次的更新。
[root@master music]# kubectl get deployments.apps -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR http-deploy 3/3 3 3 5m14s httpd httpd:2.4.39 run=apache
这是由2.4.37更新到2.4.39
--record的作用是将当前的命令记录到revision记录中，这样我们就可以知道每个revision对应的是那个配置文件了，通过
kubectl rollout history deployment 查看revision历史记录
[root@master music]# kubectl rollout history deployment deployment.apps/http-deploy REVISION CHANGE-CAUSE 1 kubectl apply --filename=httpd.yml --record=true 2 kubectl apply --filename=httpd1.yml --record=true 3 kubectl apply --filename=httpd2.yml --record=true
如果想要回到某个版本，比如说最初的2.4.37.可以执行命令
[root@master music]# kubectl rollout history deployment ##先查看一下历史版本 deployment.apps/http-deploy REVISION CHANGE-CAUSE 1 kubectl apply --filename=httpd.yml --record=true 2 kubectl apply --filename=httpd1.yml --record=true 3 kubectl apply --filename=httpd2.yml --record=true [root@master music]# kubectl get deployments.apps -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR http-deploy 3/3 3 3 21m httpd httpd:2.4.39 run=apache [root@master music]# kubectl rollout undo deployment --to-revision=1 deployment.apps/http-deploy rolled back [root@master music]# kubectl get deployments.apps -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR http-deploy 3/3 3 3 22m httpd httpd:2.4.37 run=apache
可以看到我们回到了我们制定的最开始的版本，此时。版本历史也会发生相应变化
[root@master music]# kubectl rollout history deployment deployment.apps/http-deploy REVISION CHANGE-CAUSE 2 kubectl apply --filename=httpd1.yml --record=true 3 kubectl apply --filename=httpd2.yml --record=true 4 kubectl apply --filename=httpd.yml --record=true
之前的1变成了4
Health Check
强大的自愈能力是k8s这类容器编排引擎的一个重要特性，自愈的默认实现方式是自动重启发生故障的容器，除此之外，用户还可以利用liveness和readiness探测机制设置更精细的健康检查，进而实现如下需求
1:0停机部署
2：避免部署无效的镜像
3：更加安全的滚动升级
默认的健康检查
下面我们来模拟一个容器发生故障的场景，pod配置如下
[root@master health]# cat health.yml apiVersion: v1 kind: Pod metadata: labels: test: healthcheck name: healthcheck spec: restartPolicy: OnFailure containers: - name: healthcheck image: busybox args: - /bin/bash - -c - sleep 10;exit 1
pod的restartpolicy 设置为onfailure，默认为always
sleep10；exit1 模拟容器启动10秒后发生故障
执行创建pod 命名为healthcheck
[root@master health]# kubectl get pods NAME READY STATUS RESTARTS AGE healthcheck 0/1 CrashLoopBackOff 6 7m37s
可见容器已经启动了6次
liveness探测
liveness探测让用户可以自定义判断容器是否健康的条件，如果探测失败，k8s就会重启容器
案例
[root@master health]# cat liveness.yml apiVersion: v1 kind: Pod metadata: labels: test: liveness name: liveness spec: restartPolicy: OnFailure containers: - name: liveness image: busybox args: - /bin/sh - -c - touch /tmp/healthy;sleep 30;rm -rf /tmp/healthy;sleep 600 livenessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 10 periodSeconds: 5
执行以后进程是首先创建文件/tmp/healthy,30秒以后删除，如果文件存在则健康，否则就会认为是故障
可以通过查看日志
kubectl describe pod liveness
[root@master health]# kubectl describe pod liveness Name: liveness Namespace: default Priority: 0 Node: node2/192.168.172.136 Start Time: Mon, 20 Jul 2020 22:01:31 +0800 Labels: test=liveness Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"test":"liveness"},"name":"liveness","namespace":"default"},"spec":... Status: Running IP: 10.244.2.50 IPs: IP: 10.244.2.50 Containers: liveness: Container ID: docker://5a535ca4965f649b90161b72521c4bc75c52097f7a6f0f816dee991a0000156e Image: busybox Image ID: docker-pullable://busybox@sha256:9ddee63a712cea977267342e8750ecbc60d3aab25f04ceacfa795e6fce341793 Port: <none> Host Port: <none> Args: /bin/sh -c touch /tmp/healthy;sleep 30;rm -rf /tmp/healthy;sleep 600 State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 137 Started: Mon, 20 Jul 2020 22:10:13 +0800 Finished: Mon, 20 Jul 2020 22:11:27 +0800 Ready: False Restart Count: 6 Liveness: exec [cat /tmp/healthy] delay=10s timeout=1s period=5s #success=1 #failure=3 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-ptz8b (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: default-token-ptz8b: Type: Secret (a volume populated by a Secret) SecretName: default-token-ptz8b Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 12m default-scheduler Successfully assigned default/liveness to node2 Normal Pulled 9m43s (x3 over 12m) kubelet, node2 Successfully pulled image "busybox" Normal Created 9m43s (x3 over 12m) kubelet, node2 Created container liveness Normal Started 9m43s (x3 over 12m) kubelet, node2 Started container liveness Normal Killing 8m58s (x3 over 11m) kubelet, node2 Container liveness failed liveness probe, will be restarted Normal Pulling 8m28s (x4 over 12m) kubelet, node2 Pulling image "busybox" Warning Unhealthy 7m48s (x10 over 12m) kubelet, node2 Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory Warning BackOff 2m50s (x4 over 3m3s) kubelet, node2 Back-off restarting failed container
[root@master health]# kubectl get pods NAME READY STATUS RESTARTS AGE liveness 1/1 Running 0 27s
Readiness探测
liveness探测可以告诉kubernetes什么时候通过重启容器实现自愈，readiness探测告诉kubernetes什么时候可以将容器加入到service负载均衡池中，对外提供服务
[root@master health]# cat readiness.yml apiVersion: v1 kind: Pod metadata: labels: test: readiness name: readiness spec: restartPolicy: OnFailure containers: - name: readiness image: busybox args: - /bin/sh - -c - touch /tmp/healthy;sleep 30;rm -rf /tmp/healthy;sleep 600 readinessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 10 periodSeconds: 5
查看：
[root@master health]# kubectl get pods NAME READY STATUS RESTARTS AGE liveness 1/1 Running 6 15h readiness 0/1 Running 0 20s
[root@master health]# kubectl describe pod readiness Name: readiness Namespace: default Priority: 0 Node: node2/192.168.172.136 Start Time: Tue, 21 Jul 2020 14:12:41 +0800 Labels: test=readiness Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"test":"readiness"},"name":"readiness","namespace":"default"},"spec... Status: Running IP: 10.244.2.52 IPs: IP: 10.244.2.52 Containers: readiness: Container ID: docker://22465a3dd79db1bead14ec352348c544c76a5d1b7808882b328563531f32317b Image: busybox Image ID: docker-pullable://busybox@sha256:9ddee63a712cea977267342e8750ecbc60d3aab25f04ceacfa795e6fce341793 Port: <none> Host Port: <none> Args: /bin/sh -c touch /tmp/healthy;sleep 30;rm -rf /tmp/healthy;sleep 600 State: Running Started: Tue, 21 Jul 2020 14:12:58 +0800 Ready: False Restart Count: 0 Readiness: exec [cat /tmp/healthy] delay=10s timeout=1s period=5s #success=1 #failure=3 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-ptz8b (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: default-token-ptz8b: Type: Secret (a volume populated by a Secret) SecretName: default-token-ptz8b Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 87s default-scheduler Successfully assigned default/readiness to node2 Normal Pulling 86s kubelet, node2 Pulling image "busybox" Normal Pulled 70s kubelet, node2 Successfully pulled image "busybox" Normal Created 70s kubelet, node2 Created container readiness Normal Started 70s kubelet, node2 Started container readiness Warning Unhealthy 0s (x8 over 35s) kubelet, node2 Readiness probe failed: cat: can't open '/tmp/healthy': No such file or directory
下面对Liveness探测和Readiness探测做个比较:
Liveness探测和Readiness探测是两种Health Check机制,如果不特意配置, Kubernetes将对两种探测采取相同的默认行为,即通过判断容器启动进程的返回值是否为零来判断探测是否成功。
两种探测的配置方法完全一样,支持的配置參数也一样。不同之处在于探测失败后的行为: Liveness探测是重启容器;Readiness探测则是将容器设置为不可用,不接收Service转发的请求。
Liveness探测和Readiness探测是独立执行的,二者之间没有依赖,所以可以单独使用,也可以同时使用。用Liveness探测判断容器是否需要重启以实现自愈;用Readiness探测判断容器是否已经准备好对外提供服务。
在 Scale up 中使用 health check
对于多副本应用,当执行Scale Up操作时,新副本会作为backend被添加到Service的负责均衡中,与已有副本一起处理客户的请求。考虑到应用启动通常都需要一个准备阶段,比如加载缓存数据,连接数据库等,从容器启动到正真能够提供服务是需要一段时间的。我们可以通过Readiness探测判断容器是否就绪,避免将请求发送到还没有ready的backend.
案例：
[root@master music]# cat server1.yml apiVersion: v1 kind: Service metadata: name: web-svc spec: selector: run: httpd ports: - protocol: TCP port: 8080 targetPort: 80
[root@master music]# cat web-svc.yml apiVersion: apps/v1 kind: Deployment metadata: labels: app: web name: web spec: replicas: 3 selector: matchLabels: app: web template: metadata: labels: app: web spec: containers: - image: httpd name: httpd ports: - containerPort: 8080 readinessProbe: httpGet: scheme: HTTP path: /healthy port: 8080 initialDelaySeconds: 10 periodSeconds: 5
上面配置的作用是:
容器启动10秒之后开始探测。如果http://container-ip]:8080/healthy返回代码不是200-400,表示容器没有就绪,不接收Service web-svc的请求。
每隔5秒再探测一次。
直到返回代码为200-400,表明容器已经就绪,然后将其加入到web-svc的负载均衡中,开始处理客户请求。探测会继续以5秒的间隔执行,如果连续发生3次失败,容器又会从负载均衡中移除,直到下次探测成功重新加入。
在滚动更新中使用health check
在滚动更新的时候正常副本需要10秒的时间准备工作，在此之间无法响应业务需求，如果因为人为配置错误，导致副本无法启动，那么是一个很严重的后果，在滚动更新中使用健康检查就是为了防止发生错误，新的副本只有通过readiness探测才会被添加到service如果没有通过将不会全部更换副本，业务还是可以正常运行
案例：
[root@master music]# cat web-svc.yml apiVersion: apps/v1 kind: Deployment metadata: labels: app: app name: app spec: replicas: 10 selector: matchLabels: run: app template: metadata: labels: run: app spec: containers: - image: busybox name: app args: - /bin/sh - -c - sleep 10;touch /tmp/healthy;sleep 30000 readinessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 10 periodSeconds: 5
[root@master music]# kubectl get deployments.app NAME READY UP-TO-DATE AVAILABLE AGE app 10/10 10 10 100s
[root@master music]# kubectl get pods NAME READY STATUS RESTARTS AGE app-79fc8699cd-2c6tt 1/1 Running 0 3m7s app-79fc8699cd-5svr7 1/1 Running 0 3m7s app-79fc8699cd-97hml 1/1 Running 0 3m7s app-79fc8699cd-bnbzh 1/1 Running 0 3m7s app-79fc8699cd-czghg 1/1 Running 0 3m7s app-79fc8699cd-j9dxt 1/1 Running 0 3m7s app-79fc8699cd-ltqwx 1/1 Running 0 3m7s app-79fc8699cd-prv8f 1/1 Running 0 3m7s app-79fc8699cd-wh4nv 1/1 Running 0 3m7s app-79fc8699cd-znpws 1/1 Running 0 3m7s
通过readiness检测后每10秒会启动一个副本
接下来滚动更新应用
[root@master music]# cat caoyi.yml apiVersion: apps/v1 kind: Deployment metadata: labels: app: app name: app spec: strategy: rollingUpdate: maxSurge: 35% maxUnavailable: 35% replicas: 10 selector: matchLabels: run: app template: metadata: labels: run: app spec: containers: - image: busybox name: app args: - /bin/sh - -c - sleep 30000 readinessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 10 periodSeconds: 5
查看
[root@master music]# kubectl get deployments.apps NAME READY UP-TO-DATE AVAILABLE AGE app 7/10 7 7 4m15s
新副本中不存在/tmp/healthy，所以是无法通过检测的
[root@master music]# kubectl describe deployment app Name: app Namespace: default CreationTimestamp: Tue, 21 Jul 2020 15:51:18 +0800 Labels: app=app Annotations: deployment.kubernetes.io/revision: 2 kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"kubernetes.io/change-cause":"kubectl apply --filename=caoyi.yml --... kubernetes.io/change-cause: kubectl apply --filename=caoyi.yml --record=true Selector: run=app Replicas: 10 desired | 7 updated | 14 total | 7 available | 7 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 35% max unavailable, 35% max surge Pod Template: Labels: run=app Containers: app: Image: busybox Port: <none> Host Port: <none> Args: /bin/sh -c sleep 30000 Readiness: exec [cat /tmp/healthy] delay=10s timeout=1s period=5s #success=1 #failure=3 Environment: <none> Mounts: <none> Volumes: <none> Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True ReplicaSetUpdated OldReplicaSets: app-79fc8699cd (7/7 replicas created) NewReplicaSet: app-77cdb45995 (7/7 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 6m37s deployment-controller Scaled up replica set app-79fc8699cd to 10 Normal ScalingReplicaSet 4m30s deployment-controller Scaled up replica set app-77cdb45995 to 4 Normal ScalingReplicaSet 4m30s deployment-controller Scaled down replica set app-79fc8699cd to 7 Normal ScalingReplicaSet 4m30s deployment-controller Scaled up replica set app-77cdb45995 to 7
maxSurge
此参数控制滚动更新过程中副本总数的超过DESIRED的上限。maxSurge可以是具体的整数(比如3) ,也可以是百分百,向上取整。maxSurge默认值为25%在上面的例子中, DESIRED为10,那么副本总数的最大值为:roundUp(10 + 10 * 25%) = 13所以我们看到CURRENT就是13
maxUnavailable
此参数控制滚动更新过程中,不可用的副本相占DESIRED的最大比例。maxUnavailable可以是具体的整数(比如3) ,也可以是百分百,向下取整。maxUnavailable默认值为25%
回滚
先查看一下历史版本：
[root@master music]# kubectl rollout history deployment app deployment.apps/app REVISION CHANGE-CAUSE 1 kubectl apply --filename=web-svc.yml --record=true 2 kubectl apply --filename=caoyi.yml --record=true
然后回滚：
[root@master music]# kubectl rollout undo deployment app --to-revision=1 deployment.apps/app rolled back
查看：
[root@master music]# kubectl get pods NAME READY STATUS RESTARTS AGE app-79fc8699cd-27klw 1/1 Running 0 16m app-79fc8699cd-d8ggf 1/1 Running 0 58s app-79fc8699cd-gzl5p 1/1 Running 0 16m app-79fc8699cd-mwt67 1/1 Running 0 16m app-79fc8699cd-n65dj 1/1 Running 0 58s app-79fc8699cd-rjsc9 1/1 Running 0 58s app-79fc8699cd-v4lg8 1/1 Running 0 16m app-79fc8699cd-wnlhj 1/1 Running 0 16m app-79fc8699cd-wxvc2 1/1 Running 0 16m app-79fc8699cd-zp5r4 1/1 Running 0 16m
可以看到已经回滚到第一个版本