K8S etcd集群部署

一、etcd简介

etcd是一个开源的（高可用）的分布式键值(key-value)数据库。etcd内部采用raft协议作为一致性算法，etcd基于Go语言实现。
etcd是一个服务发现系统，具备以下的特点：
　　简单：安装配置简单，而且提供了HTTP API进行交互，使用也很简单
　　安全：支持SSL证书验证
　　快速：根据官方提供的benchmark数据，单实例支持每秒2k+读操作
　　可靠：采用raft算法，实现分布式系统数据的可用性和一致性

 二、etcd在k8s中的作用

Etcd是Kubernetes集群中的一个十分重要的组件，用于保存集群所有的网络配置和对象的状态信息。
整个kubernetes系统中一共有两个服务需要用到etcd用来协同和存储配置，分别是：
　　1、网络插件flannel、对于其它网络插件也需要用到etcd存储网络的配置信息
　　2、kubernetes本身，包括各种对象的状态和元信息配置

自签证书（1、etcd集群之间要互相通信 所以要证书一样，2、flannel要把网络数据传到etcd，3、kube-apiserver：Kubernetes API，集群的统一入口，各组件协调者，以RESTful API提供接口服务，所有对象资源的增删改查和监听操作都交给APIServer处理后再提交给Etcd存储。）

三、etcd安装

etcd在生产环境中一般推荐集群方式部署。

集群的节点个数和容错：官方推荐的集群个数为奇数个，如图当节点为3个和为4个时的容错都是1, 节点5个和6个时，容错为2...

 集群的节点越多，容错性会越强，但是数据的同步份数也会越多，写性能会变差一些。合理的集群大小，就是平衡容错性和可写性。

因为etcd是go语言编写的，安装只需要下载对应的二进制文件，并放到合适的路径就行。

准备三台服务器　　

服务器信息

10.192.27.100 k8s集群master01节点	etcd01
10.192.27.115 k8s集群node01节点	etcd02
10.192.27.116 k8s集群node02节点	etcd03

[root@localhost ~]# hostnamectl set-hostname  master01
[root@master01 ~]# su -
[root@localhost ~]# hostnamectl set-hostname  node01
[root@node01 ~]# su -
[root@localhost ~]# hostnamectl set-hostname  node02
[root@node02 ~]# su -

Linux下su与su -命令的本质区别

1、防火墙 安全策略设置

防火墙设置
#停止firewall服务 systemctl stop firewalld.service
#禁止firewall开机启动 systemctl disable firewalld.service

关闭selinux
#修改配置文件 vi /etc/selinux/config
#SELINUX=enforcing #注释掉
#SELINUXTYPE=targeted #注释掉
SELINUX=disabled #增加
:wq! #保存退出
#使配置立即生效 setenforce 0 

2、生成证书

Master01 10.192.27.100配置

[root@master01 ~]# mkdir k8s
[root@master01 ~]# cd k8s/
[root@master01 k8s]# vim cfssl.sh  #新建脚本来下载cfssl软件

curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

[root@master01 k8s]# bash cfssl.sh #执行脚本
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9.8M  100  9.8M    0     0   526k      0  0:00:19  0:00:19 --:--:--  715k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2224k  100 2224k    0     0   331k      0  0:00:06  0:00:06 --:--:--  529k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6440k  100 6440k    0     0   322k      0  0:00:19  0:00:19 --:--:--  504k

[root@master01 k8s]# mkdir etcd-cert
[root@master01 k8s]# cd etcd-cert/
#新建一个生成ca.pem(证书)、ca.csr(证书签名请求,用于交叉签名或重新签名)、ca-key.pem(CA私钥,需妥善保管)、服务端证书和私钥的脚本
[root@master01 etcd-cert]# vim etcd-cert.sh 

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json <<EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#-----------------------

cat > server-csr.json <<EOF
{
    "CN": "etcd",
    "hosts": [
    "10.192.27.110",
    "10.192.27.115",
    "10.192.27.116"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

etcd-cert.sh脚本分步解释
1、创建配置文件：CA 配置文件用于配置根证书的使用场景 (profile) 和具体参数 (usage，过期时间、服务端认证、客户端认证、加密等)，后续在签名其它证书时需要指定特定场景。

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

2、创建证书签名请求文件

cat > ca-csr.json <<EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

[root@master01 etcd-cert]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -           #初始化ca
2019/11/07 11:36:55 [INFO] generating a new CA key and certificate from CSR
2019/11/07 11:36:55 [INFO] generate received request
2019/11/07 11:36:55 [INFO] received CSR
2019/11/07 11:36:55 [INFO] generating key: rsa-2048
2019/11/07 11:36:55 [INFO] encoded CSR
2019/11/07 11:36:55 [INFO] signed certificate with serial number 719641592270177579672200196043431635518255786248
[root@master01 etcd-cert]# ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem etcd-cert.sh
[root@master01 etcd-cert]#

[root@master01 etcd-cert]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca - # 初始化ca
该命令会生成运行CA所必需的文件: ca.pem(证书)、ca.csr(证书签名请求,用于交叉签名或重新签名)、ca-key.pem(CA私钥,需妥善保管)

 3、生成服务端证书和私钥

cat > server-csr.json <<EOF
{
    "CN": "etcd",
    "hosts": [
    "10.192.27.100",
    "10.192.27.115",
    "10.192.27.116"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF

[root@master01 etcd-cert]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

gencert: 生成新的key(密钥)和签名证书
    -ca：指明ca的证书
    -ca-key：指明ca的私钥文件
    -config：指明请求证书的json文件
    -profile：与-config中的profile对应，是指根据config中的profile段来生成证书的相关信息

[root@master01 etcd-cert]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2019/11/07 11:42:42 [INFO] generate received request
2019/11/07 11:42:42 [INFO] received CSR
2019/11/07 11:42:42 [INFO] generating key: rsa-2048
2019/11/07 11:42:43 [INFO] encoded CSR
2019/11/07 11:42:43 [INFO] signed certificate with serial number 36768623032790475444795305500388041408499592005
2019/11/07 11:42:43 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@master01 etcd-cert]# ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem etcd-cert.sh server.csr server-csr.json server-key.pem server.pem
[root@master01 etcd-cert]# 

ca.csr            #ca证书签名请求
ca.pem            #ca证书（公钥）
ca-key.pem      #ca私钥

server.csr        #server证书签名请求
server.pem        #server证书（公钥）
server-key.pem    #server私钥

3、二进制包安装

[root@master01 etcd-cert]# cd ..
[root@master01 k8s]# wget https://github.com/etcd-io/etcd/releasesdownload/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz  
cfssl.sh  etcd-cert  etcd-v3.3.10-linux-amd64.tar.gz 
[root@master01 k8s]# tar -xf etcd-v3.3.10-linux-amd64.tar.gz
[root@master01 k8s]# ls
cfssl.sh  etcd-cert  etcd-v3.3.10-linux-amd64  etcd-v3.3.10-linux-amd64.tar.gz 
[root@master01 k8s]# mkdir /opt/etcd/{cfg,bin,ssl} -p
[root@master01 k8s]#  ls etcd-v3.3.10-linux-amd64
Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md
[root@master01 k8s]# mv etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl /opt/etcd/bin/ #移动etcd可执行文件
[root@master01 k8s]# cp etcd-cert/*.pem /opt/etcd/ssl/  #移动证书
[root@master01 k8s]# cfssl-certinfo -cert /opt/etcd/ssl/server.pem   查看生成证书信息

4、配置文件生成

新建一个配置文件脚本  主要生成etcd的 配置文件和启动文件

[root@master01 k8s]# vim etcd.sh

#!/bin/bash
# example: ./etcd.sh etcd01 192.168.1.10 etcd02=https://192.168.1.11:2380,etcd03=https://192.168.1.12:2380

ETCD_NAME=$1
ETCD_IP=$2
ETCD_CLUSTER=$3

WORK_DIR=/opt/etcd

cat <<EOF >$WORK_DIR/cfg/etcd
#[Member]
ETCD_NAME="${ETCD_NAME}"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"  

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

cat <<EOF >/usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=${WORK_DIR}/cfg/etcd
ExecStart=${WORK_DIR}/bin/etcd \
--name=\${ETCD_NAME} \
--data-dir=\${ETCD_DATA_DIR} \
--listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=\${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=${WORK_DIR}/ssl/server.pem \
--key-file=${WORK_DIR}/ssl/server-key.pem \
--peer-cert-file=${WORK_DIR}/ssl/server.pem \
--peer-key-file=${WORK_DIR}/ssl/server-key.pem \
--trusted-ca-file=${WORK_DIR}/ssl/ca.pem \
--peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd

[root@master01 k8s]# bash etcd.sh etcd01 10.192.27.100 etcd02=https://10.192.27.115:2380,etcd03=https://10.192.27.116:2380  #执行会失败 因为其它节点没有起来
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
Job for etcd.service failed because a timeout was exceeded. See "systemctl status etcd.service" and "journalctl -xe" for details.

[root@master01 k8s]# systemctl status etcd.service
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: activating (start) since 三 2019-08-28 19:27:43 CST; 47s ago
 Main PID: 7564 (etcd)
   CGroup: /system.slice/etcd.service
           └─7564 /opt/etcd/bin/etcd --name=etcd01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://10.192.27.100:2380 --listen-client-urls=https://10.192....

8月 28 19:28:28 master01.localdomain etcd[7564]: a12885640cff075e is starting a new election at term 154
8月 28 19:28:28 master01.localdomain etcd[7564]: a12885640cff075e became candidate at term 155
8月 28 19:28:28 master01.localdomain etcd[7564]: a12885640cff075e received MsgVoteResp from a12885640cff075e at term 155
8月 28 19:28:28 master01.localdomain etcd[7564]: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to a760cbd89e28c740 at term 155
8月 28 19:28:28 master01.localdomain etcd[7564]: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to def02b5c3cfc3019 at term 155
8月 28 19:28:29 master01.localdomain etcd[7564]: a12885640cff075e is starting a new election at term 155
8月 28 19:28:29 master01.localdomain etcd[7564]: a12885640cff075e became candidate at term 156
8月 28 19:28:29 master01.localdomain etcd[7564]: a12885640cff075e received MsgVoteResp from a12885640cff075e at term 156
8月 28 19:28:29 master01.localdomain etcd[7564]: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to a760cbd89e28c740 at term 156
8月 28 19:28:29 master01.localdomain etcd[7564]: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to def02b5c3cfc3019 at term 156

[root@master01 k8s]# ps -ef | grep etcd
root       7619      1  1 19:34 ?        00:00:00 /opt/etcd/bin/etcd --name=etcd01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://10.192.27.100:2380 --listen-client-urls=https://10.192.27.100:2379,http://127.0.0.1:2379 --advertise-client-urls=https://10.192.27.100:2379 --initial-advertise-peer-urls=https://10.192.27.100:2380 --initial-cluster=etcd01=https://10.192.27.100:2380,etcd02=https://10.192.27.115:2380,etcd03=https://10.192.27.116:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
root       7626   7403  0 19:34 pts/0    00:00:00 grep --color=auto etcd

[root@master01 k8s]# tailf /var/log/messages 
Aug 28 19:36:02 master01 etcd: a12885640cff075e is starting a new election at term 459
Aug 28 19:36:02 master01 etcd: a12885640cff075e became candidate at term 460
Aug 28 19:36:02 master01 etcd: a12885640cff075e received MsgVoteResp from a12885640cff075e at term 460
Aug 28 19:36:02 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to a760cbd89e28c740 at term 460
Aug 28 19:36:02 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to def02b5c3cfc3019 at term 460
Aug 28 19:36:04 master01 etcd: a12885640cff075e is starting a new election at term 460
Aug 28 19:36:04 master01 etcd: a12885640cff075e became candidate at term 461
Aug 28 19:36:04 master01 etcd: a12885640cff075e received MsgVoteResp from a12885640cff075e at term 461
Aug 28 19:36:04 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to a760cbd89e28c740 at term 461
Aug 28 19:36:04 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to def02b5c3cfc3019 at term 461
Aug 28 19:36:05 master01 etcd: a12885640cff075e is starting a new election at term 461
Aug 28 19:36:05 master01 etcd: a12885640cff075e became candidate at term 462
Aug 28 19:36:05 master01 etcd: a12885640cff075e received MsgVoteResp from a12885640cff075e at term 462
Aug 28 19:36:05 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to a760cbd89e28c740 at term 462
Aug 28 19:36:05 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to def02b5c3cfc3019 at term 462
Aug 28 19:36:05 master01 etcd: health check for peer a760cbd89e28c740 could not connect: dial tcp 10.192.27.116:2380: connect: no route to host (prober "ROUND_TRIPPER_SNAPSHOT")
Aug 28 19:36:05 master01 etcd: health check for peer a760cbd89e28c740 could not connect: dial tcp 10.192.27.116:2380: connect: no route to host (prober "ROUND_TRIPPER_RAFT_MESSAGE")
Aug 28 19:36:05 master01 etcd: health check for peer def02b5c3cfc3019 could not connect: dial tcp 10.192.27.115:2380: connect: no route to host (prober "ROUND_TRIPPER_RAFT_MESSAGE")
Aug 28 19:36:05 master01 etcd: health check for peer def02b5c3cfc3019 could not connect: dial tcp 10.192.27.115:2380: connect: no route to host (prober "ROUND_TRIPPER_SNAPSHOT")
Aug 28 19:36:07 master01 etcd: a12885640cff075e is starting a new election at term 462
Aug 28 19:36:07 master01 etcd: a12885640cff075e became candidate at term 463
Aug 28 19:36:07 master01 etcd: a12885640cff075e received MsgVoteResp from a12885640cff075e at term 463
Aug 28 19:36:07 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to a760cbd89e28c740 at term 463
Aug 28 19:36:07 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to def02b5c3cfc3019 at term 463
Aug 28 19:36:08 master01 etcd: a12885640cff075e is starting a new election at term 463
Aug 28 19:36:08 master01 etcd: a12885640cff075e became candidate at term 464
Aug 28 19:36:08 master01 etcd: a12885640cff075e received MsgVoteResp from a12885640cff075e at term 464
Aug 28 19:36:08 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to a760cbd89e28c740 at term 464
Aug 28 19:36:08 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to def02b5c3cfc3019 at term 464
Aug 28 19:36:08 master01 etcd: publish error: etcdserver: request timed out
Aug 28 19:36:09 master01 etcd: a12885640cff075e is starting a new election at term 464
Aug 28 19:36:09 master01 etcd: a12885640cff075e became candidate at term 465
Aug 28 19:36:09 master01 etcd: a12885640cff075e received MsgVoteResp from a12885640cff075e at term 465
Aug 28 19:36:09 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to a760cbd89e28c740 at term 465
Aug 28 19:36:09 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to def02b5c3cfc3019 at term 465
Aug 28 19:36:10 master01 etcd: health check for peer def02b5c3cfc3019 could not connect: dial tcp 10.192.27.115:2380: i/o timeout (prober "ROUND_TRIPPER_RAFT_MESSAGE")
Aug 28 19:36:10 master01 etcd: health check for peer a760cbd89e28c740 could not connect: dial tcp 10.192.27.116:2380: i/o timeout (prober "ROUND_TRIPPER_SNAPSHOT")
Aug 28 19:36:10 master01 etcd: health check for peer a760cbd89e28c740 could not connect: dial tcp 10.192.27.116:2380: i/o timeout (prober "ROUND_TRIPPER_RAFT_MESSAGE")
Aug 28 19:36:10 master01 etcd: health check for peer def02b5c3cfc3019 could not connect: dial tcp 10.192.27.115:2380: i/o timeout (prober "ROUND_TRIPPER_SNAPSHOT")
Aug 28 19:36:11 master01 etcd: a12885640cff075e is starting a new election at term 465
Aug 28 19:36:11 master01 etcd: a12885640cff075e became candidate at term 466
Aug 28 19:36:11 master01 etcd: a12885640cff075e received MsgVoteResp from a12885640cff075e at term 466
Aug 28 19:36:11 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to def02b5c3cfc3019 at term 466
Aug 28 19:36:11 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to a760cbd89e28c740 at term 466
Aug 28 19:36:12 master01 etcd: a12885640cff075e is starting a new election at term 466
Aug 28 19:36:12 master01 etcd: a12885640cff075e became candidate at term 467
Aug 28 19:36:12 master01 etcd: a12885640cff075e received MsgVoteResp from a12885640cff075e at term 467
Aug 28 19:36:12 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to a760cbd89e28c740 at term 467
Aug 28 19:36:12 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to def02b5c3cfc3019 at term 467
Aug 28 19:36:13 master01 etcd: a12885640cff075e is starting a new election at term 467
Aug 28 19:36:13 master01 etcd: a12885640cff075e became candidate at term 468
Aug 28 19:36:13 master01 etcd: a12885640cff075e received MsgVoteResp from a12885640cff075e at term 468
Aug 28 19:36:13 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to a760cbd89e28c740 at term 468
Aug 28 19:36:13 master01 etcd: a12885640cff075e [logterm: 1, index: 3] sent MsgVote request to def02b5c3cfc3019 at term 468
^C

配置文件和启动文件解析

[root@master01 k8s]# vim /opt/etcd/cfg/etcd #etcd的配置文件
#[Member]  #本机etcd节点信息
ETCD_NAME="etcd01" #名字
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"  #数据库的数据目录
ETCD_LISTEN_PEER_URLS="https://10.192.27.100:2380" #etcd集群相连的端口
ETCD_LISTEN_CLIENT_URLS="https://10.192.27.100:2379" #客户端请求数据相连的端口

#[Clustering] #集群信息
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.192.27.100:2380" #通告集群端口
ETCD_ADVERTISE_CLIENT_URLS="https://10.192.27.100:2379"  #通告集群数据端口
ETCD_INITIAL_CLUSTER="etcd01=https://10.192.27.100:2380,etcd02=https://10.192.27.115:2380,etcd03=https://10.192.27.116:2380"  #集群状态地址
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"  #集群之间通信的认证  token值可以任意字符串
ETCD_INITIAL_CLUSTER_STATE="new"  #new代表新建的集群  existing 代表已存在的集群 

ETCD_NAME 节点名称，默认为default，本例中三台机器分别为：etcd01,etcd02,etcd03
ETCD_DATA_DIR 服务运行数据保存的路径

ETCD_LISTEN_PEER_URLS 监听的同伴通信的地址，比如http://ip:2380，如果有多个，使用逗号分隔。需要所有节点都能够访问，所以不要使用 localhost！
ETCD_LISTEN_CLIENT_URLS   监听的客户端服务地址
ETCD_ADVERTISE_CLIENT_URLS   对外公告的该节点客户端监听地址，这个值会告诉集群中其他节点。
ETCD_INITIAL_ADVERTISE_PEER_URLS  对外公告的该节点同伴监听地址，这个值会告诉集群中其他节点

ETCD_INITIAL_CLUSTER 集群中所有节点的信息，格式为node1=http://ip1:2380,node2=http://ip2:2380,…，注意：这里的 node1 是节点的 --name 指定的名字；后面的 ip1:2380 是 --initial-advertise-peer-urls 指定的值。
ETCD_INITIAL_CLUSTER_STATE 新建集群的时候，这个值为 new；假如加入已经存在的集群，这个值为 existing。
ETCD_INITIAL_CLUSTER_TOKEN 集群的ID，多个集群的时候，每个集群的ID必须保持唯一，否则会引发不可知错误,可以访问 https://discovery.etcd.io/new 生成一个token。

[root@master01 k8s]# vim /usr/lib/systemd/system/etcd.service  #service启动文件
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd

#引用etcd配置文件里的变量
ExecStart=/opt/etcd/bin/etcd --name=${ETCD_NAME} --data-dir=${ETCD_DATA_DIR} --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} --initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} --initial-cluster-state=new  \
--cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem \  #客户端访问etcd数据库的 数字证书 与key
--peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem \  #集群间通信的数字证书  与key
--trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem  #指定CA

Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

5、/opt/etcd文件夹、配置文件和自启文件 copy到etcd02和etcd03 

[root@master01 k8s]# scp -r /opt/etcd/ root@10.192.27.115:/opt
[root@master01 k8s]# scp -r /opt/etcd/ root@10.192.27.116:/opt
[root@master01 k8s]# scp -r /usr/lib/systemd/system/etcd.service root@10.192.27.115:/usr/lib/systemd/system/  
[root@master01 k8s]# scp -r /usr/lib/systemd/system/etcd.service root@10.192.27.116:/usr/lib/systemd/system/
[root@master01 k8s]# 

6、修改node01 10.192.27.115  node02 10.192.27.116 的etcd配置文件 

[root@node01 ~]# vim /opt/etcd/cfg/etcd 
#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.192.27.115:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.192.27.115:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.192.27.115:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.192.27.115:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.192.27.100:2380,etcd02=https://10.192.27.115:2380,etcd03=https://10.192.27.116:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

[root@node02 ~]# vim /opt/etcd/cfg/etcd 
#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.192.27.116:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.192.27.116:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.192.27.116:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.192.27.116:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.192.27.100:2380,etcd02=https://10.192.27.115:2380,etcd03=https://10.192.27.116:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

7、三台要同步时间 证书要求时间要一致 不能相差太多 与互联网时间（没有互联网 使用其它方法：https://www.cnblogs.com/linux985/p/11811951.html）

[root@master01 ~]# ntpdate time.windows.com  #三台都要执行
28 Aug 20:07:17 ntpdate[7680]: adjust time server 20.189.79.72 offset -0.006729 sec
[root@master01 ~]# date
2019年 08月 28日 星期三 20:07:24 CST

 

8、服务启动

#剩下的两台两个node节点也要执行   
[root@node01 ~]# systemctl daemon-reload
[root@node01 ~]# systemctl enable etcd
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
[root@node01 ~]# systemctlstart etcd

#master01节点重启服务
[root@master01 ~]# systemctl restart etcd
[root@master01 ~]# 

9、检查集群

[root@master01 k8s]# tailf /var/log/messages  #没有报错就成功了
Aug 28 20:12:50 master01 etcd: established a TCP streaming connection with peer def02b5c3cfc3019 (stream MsgApp v2 writer)
Aug 28 20:12:50 master01 etcd: established a TCP streaming connection with peer def02b5c3cfc3019 (stream Message writer)
Aug 28 20:12:50 master01 etcd: established a TCP streaming connection with peer def02b5c3cfc3019 (stream MsgApp v2 reader)
Aug 28 20:12:50 master01 etcd: a12885640cff075e initialzed peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)
Aug 28 20:12:50 master01 etcd: published {Name:etcd01 ClientURLs:[https://10.192.27.100:2379]} to cluster 39c79200dcc448fc
Aug 28 20:12:50 master01 etcd: ready to serve client requests
Aug 28 20:12:50 master01 etcd: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
Aug 28 20:12:50 master01 etcd: ready to serve client requests
Aug 28 20:12:50 master01 etcd: serving client requests on 10.192.27.100:2379
Aug 28 20:12:50 master01 systemd: Started Etcd Server.

测试一下集群是否健康

[root@master01 k8s]# cd etcd-cert/
[root@master01 etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://10.192.27.100:2379,https://10.192.27.115:2379,https://10.192.27.116:2379" cluster-health
member a12885640cff075e is healthy: got healthy result from https://10.192.27.100:2379
member a760cbd89e28c740 is healthy: got healthy result from https://10.192.27.116:2379
member def02b5c3cfc3019 is healthy: got healthy result from https://10.192.27.115:2379
cluster is healthy