• 处理K8S PVC删除后pod报错


    报错如下

    Jun 19 17:15:18 node1 kubelet[1722]: E0619 17:15:18.381558    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
    Jun 19 17:15:18 node1 kubelet[1722]: E0619 17:15:18.581422    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
    Jun 19 17:15:18 node1 kubelet[1722]: E0619 17:15:18.781432    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
    Jun 19 17:15:18 node1 kubelet[1722]: E0619 17:15:18.981401    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
    Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.181612    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
    Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.381434    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
    Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.581538    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
    Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.781372    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
    Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.981466    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
    Jun 19 17:15:20 node1 kubelet[1722]: E0619 17:15:20.182079    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
    Jun 19 17:15:20 node1 kubelet[1722]: E0619 17:15:20.381529    1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
    

    处理办法

    查找到该pod详情,将控制器删除即可

    [root@master ~]# kubectl get po -A|grep gr8333e7
    672f06d5992f4b4580ae04289e33dde4   gr8333e7-0
    
    [root@master ~]# kubectl describe po gr8333e7-0  -n 672f06d5992f4b4580ae04289e33dde4
    Name:         gr8333e7-0
    Namespace:    672f06d5992f4b4580ae04289e33dde4
    Priority:     0
    Node:         node1/172.31.200.68
    Start Time:   Fri, 19 Jun 2020 14:54:34 +0800
    Labels:       controller-revision-hash=gr8333e7-595ff46986
                  creater_id=1592549674520418745
    Annotations:  rainbond.com/tolerate-unready-endpoints: true
    Status:       Running
    IP:           10.244.3.196
    IPs:
      IP:           10.244.3.196
    Controlled By:  StatefulSet/gr8333e7
    Containers:
      f6719b6d0f2adace1d930dc5f48333e7:
        Container ID:   docker://2258ee0b766f2ce261563de4bda331f8bcb172ec474f0c78a9e0627eb6dbe708
        Image:          goodrain.me/f6719b6d0f2adace1d930dc5f48333e7:20200619145017
        Image ID:       docker-pullable://goodrain.me/0b8f5af437254bb55b4d8907a0bbb3ab@sha256:8e4eca55761ebadacc6503acced877fa69689389b27c56a15ba165810e563e31
        Port:           3306/TCP
        Host Port:      0/TCP
        State:          Running
          Started:      Fri, 19 Jun 2020 14:54:36 +0800
        Ready:          True
        Restart Count:  0
        Limits:
          cpu:     1280m
          memory:  1Gi
        Requests:
          cpu:      240m
          memory:   1Gi
        Readiness:  tcp-socket :3306 delay=4s timeout=5s period=3s #success=1 #failure=3
        Environment:
          LOGGER_DRIVER_NAME:      streamlog
          REVERSE_DEPEND_SERVICE:  gr512123:28d93ce6688d13325dc7986169512123,gr58ee27:599b46254ee0690d3ee750b5ab58ee27,gr5efe93:9c69edab427540f0aecc9bd0bb5efe93
          DB_HOST:                 127.0.0.1
          DB_PORT:                 3306
          TENANT_ID:               672f06d5992f4b4580ae04289e33dde4
          SERVICE_ID:              f6719b6d0f2adace1d930dc5f48333e7
          MEMORY_SIZE:             large
          SERVICE_NAME:            gr8333e7
          SERVICE_POD_NUM:         1
          HOST_IP:                  (v1:status.hostIP)
          POD_IP:                   (v1:status.podIP)
        Mounts:
          /var/lib/mysql from manual165 (rw)
          /var/run/secrets/kubernetes.io/serviceaccount from default-token-lctjg (ro)
    Conditions:
      Type              Status
      Initialized       True 
      Ready             True 
      ContainersReady   True 
      PodScheduled      True 
    Volumes:
      manual165:
        Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
        ClaimName:  manual165-gr8333e7-0
        ReadOnly:   false
      default-token-lctjg:
        Type:        Secret (a volume populated by a Secret)
        SecretName:  default-token-lctjg
        Optional:    false
    QoS Class:       Burstable
    Node-Selectors:  <none>
    Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                     node.kubernetes.io/unreachable:NoExecute for 300s
    Events:          <none>
    
    [root@master ~]# grctl service get gr8333e7 -t npwrtv4l
    Namespace:      	672f06d5992f4b4580ae04289e33dde4
    ServiceID:      	f6719b6d0f2adace1d930dc5f48333e7
    ReplicationType:	statefulset                     
    ReplicationID:  	gr8333e7                        
    Status:         	running                         
    ------------Service------------
    +---------------------+----------------+------------+
    | Name                | IP             | Port       |
    +---------------------+----------------+------------+
    | gr8333e7            | None           | (TCP:3306) |
    | service-392-3306    | 10.108.223.74  | (TCP:3306) |
    | service-392-3306out | 10.105.125.132 | (TCP:3306) |
    +---------------------+----------------+------------+
    
    ------------Ingress------------
    +------+------+
    | Name | Host |
    +------+------+
    +------+------+
    
    -------------------Pod_1-----------------------
    PodName:      	gr8333e7-0                                                                     
    PodStatus:    	Initialized : True  Ready : True  ContainersReady : True  PodScheduled : True  
    PodIP:        	10.244.3.196                                                                   
    PodHostIP:    	172.31.200.68                                                                  
    PodHostName:  	node1                                                                          
    PodVolumePath:	                                                                               
    PodStratTime: 	2020-06-19T14:54:34+08:00                                                      
    Containers:   	                                                                               
    +--------------+----------------------------------+-------------------------------------------------------------+------------------------------------+
    | ID           | Name                             | Image                                                       | State                              |
    +--------------+----------------------------------+-------------------------------------------------------------+------------------------------------+
    | 2258ee0b766f | f6719b6d0f2adace1d930dc5f48333e7 | goodrain.me/0b8f5af437254bb55b4d8907a0bbb3ab:20200424215058 | Running(2020-06-19T14:54:36+08:00) |
    +--------------+----------------------------------+-------------------------------------------------------------+------------------------------------+
    
    [root@master ~]# kubectl delete sts gr8333e7 -n 672f06d5992f4b4580ae04289e33dde4
    statefulset.apps "gr8333e7" deleted
    

    查看日志即正常

  • 相关阅读:
    hdu 3440 House Man
    hdu 2018 母牛的故事
    poj 1639 Picnic Planning 度限制mst
    uva 10870
    矩阵快速幂 模板与简单讲解
    1118sync_binlog innodb_flush_log_at_trx_commit 浅析
    1117Mysql prepare预处理语句
    1116Xlinux初学习之正则表达式和通配符
    1111分析存储引擎
    1111MySQL配置参数详解
  • 原文地址:https://www.cnblogs.com/Aaron-23/p/13164299.html
Copyright © 2020-2023  润新知