二进制部署k8s集群（二）： 签发etcd证书，安装etcd集群

【前期准备】

下载 etcd 二进制安装包：https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

下载 kubernetes 1.18.3 二进制安装包：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1183 

注意：打开链接有很多下载包，包含kubernetes-client 、kubernetes-server 、kubernetes-node ，下载其中一个64位的就行。

安装证书签发工具cfssl

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64

cp cfssl_linux-amd64 /usr/local/bin/cfssl

cp cfssljson_linux-amd64 /usr/local/bin/cfssljson

cp cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

下载etcd安装包

下载etcd-v3.4.9二进制包

下载etcd-v.3.4.9二进制安装包，

其它版本下载地址： https://github.com/etcd-io/etcd/tags 

wget  https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

 

下载完成后，解压etcd安装包，复制解压目录里的etcd与etcdctl 两个文件到 /usr/local/bin 目录下面，并且赋予两个文件可执行权限

tar xfv etcd-v3.4.9-linux-amd64.tar.gz
cd etcd-v3.4.9-linux-amd64
cp etcd /usr/local/bin 
cp etcdctl /usr/local/bin
#赋予执行权限
chmod +x /usr/local/bin/etcd
chmod +x /usr/local/bin/etcdctl

签发etcd证书

创建证书存放目录。

mkdir -p /opt/certs

【创建证书】

首先安装cfssl 证书制作工具，安装方法参考：https://www.cnblogs.com/yyee/p/13189331.html

在etcd01 (192.168.0.102)节点上创建证书。

(1) 创建根证书配置文件

vi  /opt/certs/ca-config.json

{
  "signing": {
    "default": {
      "expiry": "175200h"
     },
    "profiles":{
      "k8s-server": {
       "expiry": "175200h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth"
        ]
      },
      "k8s-client": {
       "expiry": "175200h",
        "usages": [
          "signing",
          "key encipherment",
          "client auth"
        ]
      },
      "k8s-server-client": {
       "expiry": "175200h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ]
      }
    }
  }
}

signing：表示该证书可用于签名其它证书（生成的 ca.pem 证书中 CA=TRUE）；

server auth：表示 client 可以用该该证书对 server 提供的证书进行验证；

client auth：表示 server 可以用该该证书对 client 提供的证书进行验证；

"expiry": "175200h"：证书有效期设置为 20 年； 

 

(2) 创建根证书请求文件 ca-csr.json

vi /opt/certs/ca-csr.json

{
  "CN": "k8s", 
  "key": {
     "algo": "rsa",
     "size": 2048
   },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing" ,
      "O": "k8s",
      "OU": "system"
     }
  ]
}

CN：Common Name：kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)，

浏览器使用该字段验证网站是否合法；

O：Organization：kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；

kube-apiserver 将提取的 User、Group 作为 RBAC 授权的用户标识； 

 

(3) 创建etcd证书请求文件etcd-peer-csr.json

vi /opt/certs/etcd-peer-csr.json

{
  "CN": "k8s-etcd",
  "hosts": [
    "192.168.0.101",
    "192.168.0.102",
    "192.168.0.103",
    "192.168.0.104",
    "192.168.0.105",
    "192.168.0.106"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "k8s",
      "OU": "system"
    }
  ]
}

  

三个json文件编辑完在之后，/opt/certs 目录有三个json文件。

  

(4) 生成 ca 证书和私钥 

cd /opt/certs
#生成 ca 证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

 

 生成了ca.csr, ca-key.pen, ca.pem三个私钥与证书文件。

(5) 生成etcd用的证书文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=k8s-server-client etcd-peer-csr.json | cfssljson -bare etcd-peer

注意，-profile=k8s-server-client，表示客户端与服务端要双向通讯。| cfssljson -bare etcd-peer 表示生成证书文件名为 etcd-peer 。

这次生成了etcd-peer.csr， etcd-peer-key.pem，etcd-peer.pem 三个文件

 最终产生这几个文件： ca-config.json， ca.csr， ca-csr.json ， ca-key.pem， ca.pem， etcd-peer.csr， etcd-peer-csr.json ，etcd-peer-key.pem，etcd-peer.pem 

  

(6) copy证书到其它两个节点 

 把ca.pem, etcd-peer.pem, etcd-peer-key.pem三个证书拷贝到etcd02与etcd03节点的【/opt/etcd/certs】目录，etcd只用到三个证书。

cd  /opt/certs
scp ca.pem  etcd-peer.pem  etcd-peer-key.pem  192.168.0.102:/opt/etcd/certs/
scp ca.pem  etcd-peer.pem  etcd-peer-key.pem  192.168.0.103:/opt/etcd/certs/

 

 

安装etcd可以使用SSL证书安装，也可以不使用SSL证书安装。

安装etcd (不使用SSL证书安装)

将etcd集群安装在三个节点上，三个实例节点信息为：

etcd实例名称	IP地址	Hostname
etcd01	192.168.0.102	yyee-centos-2
ctcd02	192.168.0.103	yyee-centos-3
etcd03	192.168.0.104	yyee-centos-4

(1) 在三个节点上创建工作目录

mkdir -p /var/lib/etcd/data

  

 (2) 编写etcd启动文件

编写 etcd01, etcd02, etcd03 三个节点的etcd启动文件，然后三个节点要同时启动才能启动etcd集群成功。

【编写 etcd01节点的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd 
 --name=etcd01 
 --data-dir=/var/lib/etcd/data 
 --listen-peer-urls=http://192.168.0.102:2380 
 --listen-client-urls=http://192.168.0.102:2379,http://127.0.0.1:2379 
 --initial-advertise-peer-urls=http://192.168.0.102:2380 
 --advertise-client-urls=http://192.168.0.102:2379,http://127.0.0.1 
 --initial-cluster=etcd01=http://192.168.0.102:2380,etcd02=http://192.168.0.103:2380,etcd03=http://192.168.0.104:2380 
 --initial-cluster-token=k8s-etcd-cluster 
 --initial-cluster-state=new"

Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

【编写 etcd02节点的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd 
 --name=etcd02 
 --data-dir=/var/lib/etcd/data 
 --listen-peer-urls=http://192.168.0.103:2380 
 --listen-client-urls=http://192.168.0.103:2379,http://127.0.0.1:2379 
 --initial-advertise-peer-urls=http://192.168.0.103:2380 
 --advertise-client-urls=http://192.168.0.103:2379,http://127.0.0.1 
 --initial-cluster=etcd01=http://192.168.0.102:2380,etcd02=http://192.168.0.103:2380,etcd03=http://192.168.0.104:2380 
 --initial-cluster-token=k8s-etcd-cluster 
 --initial-cluster-state=new"

Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

【编写 etcd03节点的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd 
 --name=etcd03 
 --data-dir=/var/lib/etcd/data 
 --listen-peer-urls=http://192.168.0.104:2380 
 --listen-client-urls=http://192.168.0.104:2379,http://127.0.0.1:2379 
 --initial-advertise-peer-urls=http://192.168.0.104:2380 
 --advertise-client-urls=http://192.168.0.104:2379,http://127.0.0.1 
 --initial-cluster=etcd01=http://192.168.0.102:2380,etcd02=http://192.168.0.103:2380,etcd03=http://192.168.0.104:2380 
 --initial-cluster-token=k8s-etcd-cluster 
 --initial-cluster-state=new"

Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target 

(3) 启动etcd

 然后三个节点要同时执行训动命令才能成功启动etcd。

systemctl daemon-reload
systemctl enable etcd

#这条命令要在三个节点上同时执行，第一个执行的节点会最多等待30秒让其它两个节点加入集群。
systemctl start etcd  

启动如果没报错的话就启动成功了，查看集群状态

etcdctl member list

 

查看etcd监听端口

netstat -tunlp | grep etcd

 

  

安装etcd (使用SSL证书安装)

将etcd集群安装在三个节点上，三个实例节点信息为：

etcd实例名称	IP地址	Hostname
etcd01	192.168.0.102	yyee-centos-2
ctcd02	192.168.0.103	yyee-centos-3
etcd03	192.168.0.104	yyee-centos-4

(1) 在三个节点上创建工作目录

mkdir -p /var/lib/etcd/data
mkdir -p /opt/certs

(2) copy证书到其它两个节点 

 把192.168.0.102:/opt/certs/  目录下的ca.pem, etcd-peer.pem, etcd-peer-key.pem三个证书文件拷贝到etcd02节点与etcd03节点上。

cd  /opt/certs
scp ca.pem  etcd-peer.pem  etcd-peer-key.pem  192.168.0.103:/opt/certs/
scp ca.pem  etcd-peer.pem  etcd-peer-key.pem  192.168.0.104:/opt/certs/

 

 (3) 编写etcd启动文件

编写 etcd01, etcd02, etcd03 三个节点的etcd启动文件，然后三个节点要同时启动才能启动etcd集群成功。

【编写 etcd01节点的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd 
 --name=etcd01 
 --data-dir=/var/lib/etcd/data 
 --listen-peer-urls=https://192.168.0.102:2380 
 --listen-client-urls=https://192.168.0.102:2379,http://127.0.0.1:2379 
 --initial-advertise-peer-urls=https://192.168.0.102:2380 
 --advertise-client-urls=https://192.168.0.102:2379 
 --initial-cluster=etcd01=https://192.168.0.102:2380,etcd02=https://192.168.0.103:2380,etcd03=https://192.168.0.104:2380 
 --initial-cluster-token=k8s-etcd-cluster 
 --initial-cluster-state=new 
 --cert-file=/opt/etcd/certs/etcd-peer.pem 
 --key-file=/opt/etcd/certs/etcd-peer-key.pem 
 --peer-cert-file=/opt/etcd/certs/etcd-peer.pem 
 --peer-key-file=/opt/etcd/certs/etcd-peer-key.pem 
 --trusted-ca-file=/opt/etcd/certs/ca.pem 
 --peer-trusted-ca-file=/opt/etcd/certs/ca.pem"


Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

【编写 etcd02节点的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd 
 --name=etcd02 
 --data-dir=/var/lib/etcd/data 
 --listen-peer-urls=https://192.168.0.103:2380 
 --listen-client-urls=https://192.168.0.103:2379,http://127.0.0.1:2379 
 --initial-advertise-peer-urls=https://192.168.0.103:2380 
 --advertise-client-urls=https://192.168.0.103:2379 
 --initial-cluster=etcd01=https://192.168.0.102:2380,etcd02=https://192.168.0.103:2380,etcd03=https://192.168.0.104:2380 
 --initial-cluster-token=k8s-etcd-cluster 
 --initial-cluster-state=new 
 --cert-file=/opt/etcd/certs/etcd-peer.pem 
 --key-file=/opt/etcd/certs/etcd-peer-key.pem 
 --peer-cert-file=/opt/etcd/certs/etcd-peer.pem 
 --peer-key-file=/opt/etcd/certs/etcd-peer-key.pem 
 --trusted-ca-file=/opt/etcd/certs/ca.pem 
 --peer-trusted-ca-file=/opt/etcd/certs/ca.pem"


Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

【编写 etcd03节点的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd 
 --name=etcd03 
 --data-dir=/var/lib/etcd/data 
 --listen-peer-urls=https://192.168.0.104:2380 
 --listen-client-urls=https://192.168.0.104:2379,http://127.0.0.1:2379 
 --initial-advertise-peer-urls=https://192.168.0.104:2380 
 --advertise-client-urls=https://192.168.0.104:2379 
 --initial-cluster=etcd01=https://192.168.0.102:2380,etcd02=https://192.168.0.103:2380,etcd03=https://192.168.0.104:2380 
 --initial-cluster-token=k8s-etcd-cluster 
 --initial-cluster-state=new 
 --cert-file=/opt/etcd/certs/etcd-peer.pem 
 --key-file=/opt/etcd/certs/etcd-peer-key.pem 
 --peer-cert-file=/opt/etcd/certs/etcd-peer.pem 
 --peer-key-file=/opt/etcd/certs/etcd-peer-key.pem 
 --trusted-ca-file=/opt/etcd/certs/ca.pem 
 --peer-trusted-ca-file=/opt/etcd/certs/ca.pem"


Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target 

(4) 启动etcd

 然后三个节点要同时执行训动命令才能成功启动etcd。

systemctl daemon-reload
systemctl enable etcd

#这条命令要在三个节点上同时执行，第一个执行的节点会最多等待30秒让其它两个节点加入集群。
systemctl start etcd  

启动如果没报错的话就启动成功了，查看集群状态

etcdctl member list

 

查看etcd监听端口

netstat -tunlp | grep etcd