• 解决The swarm does not have a leader


    1问题:

    最近有一个测试环境的Swarm集群挂了,这个集群有两个管理节点,执行 docker node ls,均报:

    The swarm does not have a leader. It's possible that too few managers are online. Make sure more than half of the managers are online

    明明两个管理节点都在线。

    2分析:

    通过docker info命令,看到一条错误信息

    Error: rpc error: code = Unknown desc = The swarm does not have a leader. It's possible that too few managers are online. Make sure more than half of the managers are online.

    逐个分析两个节点的日志,发现周期性打印的错误日志:

    第一个管理节点:

    Mar 4 09:30:05 manager1 dockerd: time="2020-03-04T09:30:05.663865244+08:00" level=error

    msg="error sending message to peer" error="rpc error: code = Internal desc = connection error: desc = "transport: x509: certificate has expired or is not yet valid""

    第二个管理节点报:

    Mar 4 09:08:01 manager2 dockerd: time="2020-03-04T09:08:01.446858105+08:00" level=warning

    msg="error renewing TLS certificate: rpc error: code = Internal desc = connection error: desc = "transport: remote error: tls: bad certificate""

    初步得出结论,第二个管理节点证书有问题,并且很大可能是过期了,

    根据字面信息猜测一下:这里好像是个BUG,刷新本地证书需要请求某一个远程节点,请求远程节点又报证书不对,形成悖论。

    查看两台机器的时间,均是正常时间

    3验证:

    通过命令

    docker swarm ca | openssl x509 -noout -text

    查看第二个管理节点证书,命令报错无法显示证书信息

    直接通过谷歌浏览器访问两个节点的2377端口 https://x.x.x.x:2377

    点击证书,查看证书,发现有效期不在当前时间范围内,接着着手更新证书有效期

    接着面临问题:证书在哪存放?怎么更新?参考了以下地址的内容:

    证书相关 github上的讨论

    4最终解决:

    管理节点二因为证书失效,直接主动让它离开集群

    docker swarm leave --force

    管理节点一仍然不正常,在管理节点一上执行命令

    docker swarm init --force-new-cluster --advertise-addr x.x.x.x

    (x.x.x.x是你服务器的IP地址)

    发现无法正常执行,重启了docker进程

    systemctl restart docker

    等待时间较长,之后再次执行

    docker swarm init --force-new-cluster --advertise-addr x.x.x.x

    集群恢复正常,并且之前的部署和配置依然存在,算是解决了问题

    (完毕)

  • 相关阅读:
    pat00-自测5. Shuffling Machine (20)
    Spiral Matrix
    Search in Rotated Sorted Array II
    Search in Rotated Sorted Array
    Best Time to Buy and Sell Stock II
    4Sum
    3Sum Closest
    3Sum
    MySQL存储过程、函数和游标
    Word Ladder
  • 原文地址:https://www.cnblogs.com/flying607/p/12407952.html
Copyright © 2020-2023  润新知