kubernetes二进制安装部署过程

一. 机器规划

IP	机器名称	机器配置	操作系统	机器角色	安装组件
172.16.2.4	k8s-master01	2c 4g	centos7.8	master	kube-apiserver、kube-controller-manager、kube-scheduler、etcd
172.16.2.5	k8s-master02	2c 4g	centos7.8	master	kube-apiserver、kube-controller-manager、kube-scheduler、etcd
172.16.2.6	k8s-master03	2c 4g	centos7.8	master	kube-apiserver、kube-controller-manager、kube-scheduler、etcd
172.16.2.7	k8s-node01	2c 4g	centos7.8	worker	kubelet、kube-proxy
172.16.2.8	k8s-node02	2c 4g	centos7.8	worker	kubelet、kube-proxy
172.16.2.9	k8s-node3	2c 4g	centos7.8	worker	kubelet、kube-proxy
172.16.2.100	VIP

注：此处VIP是采用的云厂商的SLB，你也可以使用haproxy + keepalived的方式实现。

二. 搭建集群

2.1 机器环境初始化

2.1.1 按照前面的表格修改主机名

2.1.2 配置各个机器的host解析

cat >> /etc/hosts << EOF
172.16.2.4 k8s-master01
172.16.2.5 k8s-master02
172.16.2.6 k8s-master03
172.16.2.7 k8s-node01
172.16.2.8 k8s-node02
172.16.2.9 k8s-node03
EOF

2.1.3 关闭防火墙和selinux

systemctl stop firewalld
setenforce 0
sed -i 's/^SELINUX=.\*/SELINUX=disabled/' /etc/selinux/config

2.1.4 关闭swap

swapoff -a
永久关闭，修改/etc/fstab,注释掉swap一行

2.1.5 时间同步

yum install -y chrony
systemctl start chronyd
systemctl enable chronyd
chronyc sources

2.1.6 修改内核参数

cat > /etc/sysctl.d/k8s.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

2.1.7 加载ipvs内核模块

modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
lsmod | grep ip_vs
lsmod | grep nf_conntrack_ipv4
yum install -y ipvsadm

2.2 配置工作目录

每台机器都需要配置证书文件、组件的配置文件、组件的服务启动文件，现专门选择 master1 来统一生成这些文件，然后再分发到其他机器。以下操作在 master1 上进行

mkdir -p /data/work
注：该目录为配置文件和证书文件生成目录，后面的所有文件生成相关操作均在此目录下进行
ssh-keygen -t rsa -b 2048
将秘钥分发到另外五台机器，让 master1 可以免密码登录其他机器

2.3 搭建etcd集群

2.3.1 配置etcd工作目录

mkdir -p /etc/etcd                               # 配置文件存放目录
mkdir -p /etc/etcd/ssl                           # 证书文件目录

2.3.2 创建etcd证书

创建配置文件

注：master2和master3分别修改配置文件中etcd名字和ip，并创建目录 /var/lib/etcd/default.etcd

cat etcd.conf
#[Member]
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.16.2.4:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.16.2.4:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.2.4:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.2.4:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://172.16.2.4:2380,etcd2=https://172.6.2.5:2380,etcd3=https://172.16.2.6:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

注：
ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

创建启动服务文件

方法一: 有配置文件

cat etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/etc/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
[root@k8s-master01 work]# 

方法二: 无配置文件

etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --name=etcd1 \
  --data-dir=/var/lib/etcd/default.etcd \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --listen-peer-urls=https://172.16.2.4:2380 \
  --listen-client-urls=https://172.16.2.4:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://172.16.2.4:2379 \
  --initial-advertise-peer-urls=https://172.16.2.4:2380 \
  --initial-cluster=etcd1=https://172.16.2.4:2380,etcd2=https://172.16.2.5:2380,etcd3=https://172.16.2.6:2380 \
  --initial-cluster-token=etcd-cluster \
  --initial-cluster-state=new
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

同步文件到各个节点

cp ca*.pem /etc/etcd/ssl/
cp etcd*.pem /etc/etcd/ssl/
cp etcd.conf /etc/etcd/
cp etcd.service /usr/lib/systemd/system/
for i in k8s-master02 k8s-master03;do ssh $i mkdir -pv /etc/etcd/ssl/; done
for i in k8s-master02 k8s-master03;do scp etcd.conf $i:/etc/etcd/;done
for i in k8s-master02 k8s-master03;do scp etcd*.pem ca*.pem $i:/etc/etcd/ssl/;done
for i in k8s-master02 k8s-master03;do scp etcd.service $i:/usr/lib/systemd/system/;done

启动etcd集群

mkdir -p /var/lib/etcd/default.etcd
systemctl daemon-reload
systemctl enable etcd.service
systemctl start etcd.service
systemctl status etcd

ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://172.16.2.4:2379,https://172.16.2.5:2379,https://172.16.2.6:2379 endpoint health