kubernetes排错系列：（二）、运行很久的kubernetes集群，创建出来的pod都是pending状态

1.查看pod信息

# 查看pod 报错信息
kubectl get pods
发现pod的ip没有 生成，也没有分配到某个node节点

# 查看pod详细时间
kubectl describe pods
发现pod事件为空

2.查看集群信息

kubectl get nodes

发现集群状态正常

kubectl cluster-info

Kubernetes master is running at https://xx.xx.55.113
KubeDNS is running at https://xx.xx.55.113:6443/api

3. 查看kubelet日志发现

journalctl -xefu kubelet

Sep 17 14:38:34 cbov10-sso55-113 kubelet[9892]: E0917 14:38:34.750672    9892 kuberuntime_manager.go:917] getPodContainerStatuses for pod "kube-scheduler-k8s-master_kube-system(a29556bf1d34f898bf5d0ce3c15a5948)" failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/s=13b9f39777ba4419b309b8b3ae11760e;i=121b8;b=37ab1d18c20d41b2a88be512b10aba22;m=559ecd: no such file or directory
Sep 17 14:38:35 cbov10-sso55-113 kubelet[9892]: E0917 14:38:35.260404    9892 remote_runtime.go:321] ContainerStatus "fad2baa3755429e94c4b79549ce2a2d2a2b018e1c55bf282d1b5b0be0b4022fe" from runtime service failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/s=13b9f39777ba4419b309b8b3ae11760e;i=121b8;b=37ab1d18c20d41b2a88be512b10aba22;m=559ecd: no such file or directory
Sep 17 14:38:35 cbov10-sso55-113 kubelet[9892]: E0917 14:38:35.260498    9892 kuberuntime_gc.go:142] Failed to remove container "fad2baa3755429e94c4b79549ce2a2d2a2b018e1c55bf282d1b5b0be0b4022fe": failed to get container status "fad2baa3755429e94c4b79549ce2a2d2a2b018e1c55bf282d1b5b0be0b4022fe": rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/s=13b9f39777ba4419b309b8b3ae11760e;i=121b8;b=37ab1d18c20d41b2a88be512b10aba22;m=559ecd: no such file or directory
Sep 17 14:38:44 cbov10-sso55-113 kubelet[9892]: E0917 14:38:44.674953    9892 pod_workers.go:190] Error syncing pod a29556bf1d34f898bf5d0ce3c15a5948 ("kube-scheduler-cbov10-sso55-113_kube-system(a29556bf1d34f898bf5d0ce3c15a5948)"), skipping: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/s=13b9f39777ba4419b309b8b3ae11760e;i=121b8;b=37ab1d18c20d41b2a88be512b10aba22;m=559ecd: no such file or directory

4.清理docker占用的磁盘空间

#可以用于清理磁盘，删除关闭的容器、无用的数据卷和网络
docker system prune -a

5.之后再次部署，发现pod能正常启动。