高可用简介
kubernetes高可用部署参考：
https://kubernetes.io/docs/setup/independent/high-availability/
https://github.com/kubernetes-sigs/kubespray
https://github.com/wise2c-devops/breeze
https://github.com/cookeem/kubeadm-ha
拓扑选择
配置高可用（HA）Kubernetes集群，有以下两种可选的etcd拓扑：
- 集群master节点与etcd节点共存，etcd也运行在控制平面节点上
- 使用外部etcd节点，etcd节点与master在不同节点上运行
堆叠的etcd拓扑
堆叠HA集群是这样的拓扑，其中etcd提供的分布式数据存储集群与由kubeamd管理的运行master组件的集群节点堆叠部署。
每个master节点运行kube-apiserver，kube-scheduler和kube-controller-manager的一个实例。kube-apiserver使用负载平衡器暴露给工作节点。
每个master节点创建一个本地etcd成员，该etcd成员仅与本节点kube-apiserver通信。这同样适用于本地kube-controller-manager 和kube-scheduler实例。
该拓扑将master和etcd成员耦合在相同节点上。比设置具有外部etcd节点的集群更简单，并且更易于管理复制。
但是，堆叠集群存在耦合失败的风险。如果一个节点发生故障，则etcd成员和master实例都将丢失，并且冗余会受到影响。您可以通过添加更多master节点来降低此风险。
因此，您应该为HA群集运行至少三个堆叠的master节点。
这是kubeadm中的默认拓扑。使用kubeadm init和kubeadm join --experimental-control-plane命令时，在master节点上自动创建本地etcd成员。
外部etcd拓扑
具有外部etcd的HA集群是这样的拓扑，其中由etcd提供的分布式数据存储集群部署在运行master组件的节点形成的集群外部。
像堆叠ETCD拓扑结构，在外部ETCD拓扑中的每个master节点运行一个kube-apiserver，kube-scheduler和kube-controller-manager实例。并且kube-apiserver使用负载平衡器暴露给工作节点。但是，etcd成员在不同的主机上运行，每个etcd主机与kube-apiserver每个master节点进行通信。
此拓扑将master节点和etcd成员分离。因此，它提供了HA设置，其中丢失master实例或etcd成员具有较小的影响并且不像堆叠的HA拓扑那样影响集群冗余。
但是，此拓扑需要两倍于堆叠HA拓扑的主机数。具有此拓扑的HA群集至少需要三个用于master节点的主机和三个用于etcd节点的主机。
部署要求
使用kubeadm部署高可用性Kubernetes集群的两种不同方法：
- 使用堆叠master节点。这种方法需要较少的基础设施，etcd成员和master节点位于同一位置。
- 使用外部etcd集群。这种方法需要更多的基础设施， master节点和etcd成员是分开的。
在继续之前，您应该仔细考虑哪种方法最能满足您的应用程序和环境的需求。
部署要求
- 至少3个master节点
- 至少3个worker节点
- 所有节点网络全部互通（公共或私有网络）
- 所有机器都有sudo权限
- 从一个设备到系统中所有节点的SSH访问
- 所有节点安装kubeadm和kubelet，kubectl是可选的。
- 针对外部etcd集群，你需要为etcd成员额外提供3个节点
负载均衡
部署集群前首选需要为kube-apiserver创建负载均衡器。
注意：负载平衡器有许多中配置方式。可以根据你的集群要求选择不同的配置方案。在云环境中，您应将master节点作为负载平衡器TCP转发的后端。此负载平衡器将流量分配到其目标列表中的所有健康master节点。apiserver的运行状况检查是对kube-apiserver侦听的端口的TCP检查（默认值:6443）。
负载均衡器必须能够与apiserver端口上的所有master节点通信。它还必须允许其侦听端口上的传入流量。另外确保负载均衡器的地址始终与kubeadm的ControlPlaneEndpoint地址匹配。
haproxy/nignx+keepalived是其中可选的负载均衡方案，针对公有云环境可以直接使用运营商提供的负载均衡产品。
部署时首先将第一个master节点添加到负载均衡器并使用以下命令测试连接：
# nc -v LOAD_BALANCER_IP PORT
由于apiserver尚未运行，因此预计会出现连接拒绝错误。但是，超时意味着负载均衡器无法与master节点通信。如果发生超时，请重新配置负载平衡器以与master节点通信。将剩余的master节点添加到负载平衡器目标组。
第2章 部署集群
本次使用kubeadm部署kubernetes v1.14.1高可用集群，包含3个master节点和1个node节点，部署步骤以官方文档为基础，负载均衡部分采用haproxy+keepalived容器方式实现。所有组件版本以kubernetes v1.14.1为准，其他组件以当前最新版本为准。
基本配置
节点信息：
主机名 IP地址 角色 OS CPU/MEM 磁盘 网卡 k8s-master01 192.168.92.10 master CentOS7.6 2C2G 60G x1 k8s-master02 192.168.92.11 master CentOS7.6 2C2G 60G x1 k8s-master03 192.168.92.12 master CentOS7.6 2C2G 60G x1 k8s-node01 192.168.92.13 node CentOS7.6 2C2G 60G x1 K8S VIP 192.168.92.30 – – – - -
以下操作在所有节点执行。
#配置主机名
hostnamectl set-hostname k8s-master01 hostnamectl set-hostname k8s-master02 hostnamectl set-hostname k8s-master03 hostnamectl set-hostname k8s-node01
#修改/etc/hosts
cat >> /etc/hosts << EOF 192.168.92.10 k8s-master01 192.168.92.11 k8s-master02 192.168.92.12 k8s-master03 192.168.92.13 k8s-node01 EOF
# 开启firewalld防火墙并允许所有流量
systemctl start firewalld && systemctl enable firewalld firewall-cmd --set-default-zone=trusted firewall-cmd --complete-reload
# 关闭selinux
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config && setenforce 0
#关闭swap
swapoff -a yes | cp /etc/fstab /etc/fstab_bak cat /etc/fstab_bak | grep -v swap > /etc/fstab
配置时间同步
使用chrony同步时间，centos7默认已安装，这里修改时钟源，所有节点与网络时钟源同步：
# 安装chrony： yum install -y chrony cp /etc/chrony.conf{,.bak} # 注释默认ntp服务器 sed -i 's/^server/#&/' /etc/chrony.conf # 指定上游公共 ntp 服务器 cat >> /etc/chrony.conf << EOF server 0.asia.pool.ntp.org iburst server 1.asia.pool.ntp.org iburst server 2.asia.pool.ntp.org iburst server 3.asia.pool.ntp.org iburst EOF # 设置时区 timedatectl set-timezone Asia/Shanghai # 重启chronyd服务并设为开机启动： systemctl enable chronyd && systemctl restart chronyd #验证,查看当前时间以及存在带*的行 timedatectl && chronyc sources
加载IPVS模块
在所有的Kubernetes节点执行以下脚本（若内核大于4.19替换nf_conntrack_ipv4为nf_conntrack）:
cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF #执行脚本 chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4 #安装相关管理工具 yum install ipset ipvsadm -y
配置内核参数
cat > /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_nonlocal_bind = 1 net.ipv4.ip_forward = 1 vm.swappiness=0 EOF sysctl --system
安装docker
CRI安装参考：https://kubernetes.io/docs/setup/cri/
要在Pod中运行容器，Kubernetes使用容器运行时。以下是可选的容器运行时。
- Docker
- CRI-O
- Containerd
- Other CRI runtimes: frakti
Cgroup驱动程序简介
当systemd被选为Linux发行版的init系统时，init进程会生成并使用根控制组（cgroup）并充当cgroup管理器。Systemd与cgroup紧密集成，并将为每个进程分配cgroup。可以配置容器运行时和要使用的kubelet cgroupfs。cgroupfs与systemd一起使用意味着将有两个不同的cgroup管理器。
Control groups用于约束分配给进程的资源。单个cgroup管理器将简化正在分配的资源的视图，并且默认情况下将具有更可靠的可用和使用资源视图。
当我们有两个managers时，我们最终会得到两个这些资源的视图。我们已经看到了现场的情况，其中配置cgroupfs用于kubelet和Docker systemd 的节点以及在节点上运行的其余进程在资源压力下变得不稳定。
更改设置，使容器运行时和kubelet systemd用作cgroup驱动程序，从而使系统稳定。请注意native.cgroupdriver=systemd下面Docker设置中的选项。
安装并配置docker
以下操作在所有节点执行。
# 安装依赖软件包 yum install -y yum-utils device-mapper-persistent-data lvm2 # 添加Docker repository，这里改为国内阿里云yum源 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # 安装docker-ce yum update -y && yum install -y docker-ce ## 创建 /etc/docker 目录 mkdir /etc/docker # 配置 daemon. cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ], "registry-mirrors": ["https://uyah70su.mirror.aliyuncs.com"] } EOF #注意，由于国内拉取镜像较慢，配置文件最后追加了阿里云镜像加速配置。 mkdir -p /etc/systemd/system/docker.service.d # 重启docker服务 systemctl daemon-reload && systemctl restart docker && systemctl enable docker
安装负载均衡
kubernetes master 节点运行如下组件：
- kube-apiserver
- kube-scheduler
- kube-controller-manager
kube-scheduler 和 kube-controller-manager 可以以集群模式运行，通过 leader 选举产生一个工作进程，其它进程处于阻塞模式。
kube-apiserver可以运行多个实例，但对其它组件需要提供统一的访问地址，该地址需要高可用。本次部署使用 keepalived+haproxy 实现 kube-apiserver VIP 高可用和负载均衡。
haproxy+keepalived配置vip，实现了api唯一的访问地址和负载均衡。keepalived 提供 kube-apiserver 对外服务的 VIP。haproxy 监听 VIP，后端连接所有 kube-apiserver 实例，提供健康检查和负载均衡功能。
运行 keepalived 和 haproxy 的节点称为 LB 节点。由于 keepalived 是一主多备运行模式，故至少两个 LB 节点。
本次部署复用 master 节点的三台机器，在所有3个master节点部署haproxy和keepalived组件，以达到更高的可用性，haproxy 监听的端口(6444) 需要与 kube-apiserver的端口 6443 不同，避免冲突。
keepalived 在运行过程中周期检查本机的 haproxy 进程状态，如果检测到 haproxy 进程异常，则触发重新选主的过程，VIP 将飘移到新选出来的主节点，从而实现 VIP 的高可用。
所有组件（如 kubeclt、apiserver、controller-manager、scheduler 等）都通过 VIP +haproxy 监听的6444端口访问 kube-apiserver 服务。
负载均衡架构图如下：
运行HA容器
使用的容器镜像为睿云智合开源项目breeze相关镜像，具体使用方法请访问：
https://github.com/wise2c-devops
其他选择：haproxy镜像也可以使用dockerhub官方镜像，但keepalived未提供官方镜像，可自行构建或使用dockerhub他人已构建好的镜像，本次部署全部使用breeze提供的镜像。
在3个master节点以容器方式部署haproxy，容器暴露6444端口，负载均衡到后端3个apiserver的6443端口，3个节点haproxy配置文件相同。
以下操作在master01节点执行。
创建haproxy启动脚本
编辑start-haproxy.sh文件，修改Kubernetes Master节点IP地址为实际Kubernetes集群所使用的值（Master Port默认为6443不用修改）：
mkdir -p /data/lb cat > /data/lb/start-haproxy.sh << "EOF" #!/bin/bash MasterIP1=192.168.92.10 MasterIP2=192.168.92.11 MasterIP3=192.168.92.12 MasterPort=6443 docker run -d --restart=always --name HAProxy-K8S -p 6444:6444 -e MasterIP1=$MasterIP1 -e MasterIP2=$MasterIP2 -e MasterIP3=$MasterIP3 -e MasterPort=$MasterPort wise2c/haproxy-k8s EOF
创建keepalived启动脚本
编辑start-keepalived.sh文件，修改虚拟IP地址VIRTUAL_IP、虚拟网卡设备名INTERFACE、虚拟网卡的子网掩码NETMASK_BIT、路由标识符RID、虚拟路由标识符VRID的值为实际Kubernetes集群所使用的值。（CHECK_PORT的值6444一般不用修改，它是HAProxy的暴露端口，内部指向Kubernetes Master Server的6443端口）
cat > /data/lb/start-keepalived.sh << "EOF" #!/bin/bash VIRTUAL_IP=192.168.92.30 INTERFACE=ens33 NETMASK_BIT=24 CHECK_PORT=6444 RID=10 VRID=160 MCAST_GROUP=224.0.0.18 docker run -itd --restart=always --name=Keepalived-K8S --net=host --cap-add=NET_ADMIN -e VIRTUAL_IP=$VIRTUAL_IP -e INTERFACE=$INTERFACE -e CHECK_PORT=$CHECK_PORT -e RID=$RID -e VRID=$VRID -e NETMASK_BIT=$NETMASK_BIT -e MCAST_GROUP=$MCAST_GROUP wise2c/keepalived-k8s EOF
复制启动脚本到其他2个master节点
[root@k8s-master02 ~]# mkdir -p /data/lb [root@k8s-master03 ~]# mkdir -p /data/lb [root@k8s-master01 ~]# scp start-haproxy.sh start-keepalived.sh 192.168.92.11:/data/lb/ [root@k8s-master01 ~]# scp start-haproxy.sh start-keepalived.sh 192.168.92.12:/data/lb/
分别在3个master节点运行脚本启动haproxy和keepalived容器：
sh /data/lb/start-haproxy.sh && sh /data/lb/start-keepalived.sh
验证HA状态
查看容器运行状态
[root@k8s-master01 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c1d1901a7201 wise2c/haproxy-k8s "/docker-entrypoint.…" 5 days ago Up 3 hours 0.0.0.0:6444->6444/tcp HAProxy-K8S 2f02a9fde0be wise2c/keepalived-k8s "/usr/bin/keepalived…" 5 days ago Up 3 hours Keepalived-K8S
查看网卡绑定的vip 为192.168.92.30
[root@k8s-master01 ~]# ip a | grep ens33 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.92.10/24 brd 192.168.92.255 scope global noprefixroute ens33 inet 192.168.92.30/24 scope global secondary ens33
查看监听端口为6444
[root@k8s-master01 ~]# netstat -tnlp | grep 6444 tcp6 0 0 :::6444 :::* LISTEN 11695/docker-proxy
keepalived配置文件中配置了vrrp_script脚本，使用nc命令对haproxy监听的6444端口进行检测，如果检测失败即认定本机haproxy进程异常，将vip漂移到其他节点。
所以无论本机keepalived容器异常或haproxy容器异常都会导致vip漂移到其他节点，可以停掉vip所在节点任意容器进行测试。
[root@k8s-master01 ~]# docker stop HAProxy-K8S
HAProxy-K8S
#可以看到vip漂移到k8s-master02节点
[root@k8s-master02 ~]# ip a | grep ens33 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.92.11/24 brd 192.168.92.255 scope global noprefixroute ens33 inet 192.168.92.30/24 scope global secondary ens33
也可以在本地执行该nc命令查看结果
[root@k8s-master02 ~]# yum install -y nc [root@k8s-master02 ~]# nc -v -w 2 -z 127.0.0.1 6444 2>&1 | grep 'Connected to' | grep 6444 Ncat: Connected to 127.0.0.1:6444.
关于haproxy和keepalived配置文件可以在github源文件中参考Dockerfile，或使用docker exec -it xxx sh命令进入容器查看，容器中的具体路径：
- /etc/keepalived/keepalived.conf /usr/local/etc/haproxy/haproxy.cfg
负载均衡部分配置完成后即可开始部署kubernetes集群。
安装kubeadm
以下操作在所有节点执行。
#由于官方源国内无法访问，这里使用阿里云yum源进行替换：
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF #安装kubeadm、kubelet、kubectl,注意这里默认安装当前最新版本v1.14.1: yum install -y kubeadm kubelet kubectl systemctl enable kubelet && systemctl start kubelet
初始化master节点
初始化参考：
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1
创建初始化配置文件
可以使用如下命令生成初始化配置文件
kubeadm config print init-defaults > kubeadm-config.yaml
根据实际部署环境修改信息：
[root@k8s-master01 kubernetes]# vim kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta1 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.92.10 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: k8s-master01 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta1 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: "192.168.92.30:6444" controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration kubernetesVersion: v1.14.1 networking: dnsDomain: cluster.local podSubnet: "10.244.0.0/16" serviceSubnet: 10.96.0.0/12 scheduler: {} --- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration featureGates: SupportIPVSProxyMode: true mode: ipvs
配置说明：
- controlPlaneEndpoint：为vip地址和haproxy监听端口6444
- imageRepository:由于国内无法访问google镜像仓库k8s.gcr.io，这里指定为阿里云镜像仓库registry.aliyuncs.com/google_containers
- podSubnet:指定的IP地址段与后续部署的网络插件相匹配，这里需要部署flannel插件，所以配置为10.244.0.0/16
- mode: ipvs:最后追加的配置为开启ipvs模式。
在集群搭建完成后可以使用如下命令查看生效的配置文件：
kubectl -n kube-system get cm kubeadm-config -oyaml
初始化Master01节点
这里追加tee命令将初始化日志输出到kubeadm-init.log中以备用（可选）。
kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs | tee kubeadm-init.log
该命令指定了初始化时需要使用的配置文件，其中添加–experimental-upload-certs参数可以在后续执行加入节点时自动分发证书文件。
初始化示例
[root@k8s-master01 ~]# kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs | tee kubeadm-init.log [init] Using Kubernetes version: v1.14.1 [preflight] Running pre-flight checks [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Activating the kubelet service [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.92.10 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.92.10 127.0.0.1 ::1] [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "ca" certificate and key [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.92.10 192.168.92.30] [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "admin.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "kubelet.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "controller-manager.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 19.020444 seconds [upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Storing the certificates in ConfigMap "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: 11def25d624a2150b57715e21b0c393695bc6a70d932e472f75d24f747eb657e [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: abcdef.0123456789abcdef [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: kubeadm join 192.168.92.30:6444 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:7b232b343577bd5fac312996b9fffb3c88f8f8bb39f46bf865ac9f9f52982b82 --experimental-control-plane --certificate-key 11def25d624a2150b57715e21b0c393695bc6a70d932e472f75d24f747eb657e Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --experimental-upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.92.30:6444 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:7b232b343577bd5fac312996b9fffb3c88f8f8bb39f46bf865ac9f9f52982b82
kubeadm init主要执行了以下操作：
- [init]：指定版本进行初始化操作
- [preflight] ：初始化前的检查和下载所需要的Docker镜像文件
- [kubelet-start]：生成kubelet的配置文件”/var/lib/kubelet/config.yaml”，没有这个文件kubelet无法启动，所以初始化之前的kubelet实际上启动失败。
- [certificates]：生成Kubernetes使用的证书，存放在/etc/kubernetes/pki目录中。
- [kubeconfig] ：生成 KubeConfig 文件，存放在/etc/kubernetes目录中，组件之间通信需要使用对应文件。
- [control-plane]：使用/etc/kubernetes/manifest目录下的YAML文件，安装 Master 组件。
- [etcd]：使用/etc/kubernetes/manifest/etcd.yaml安装Etcd服务。
- [wait-control-plane]：等待control-plan部署的Master组件启动。
- [apiclient]：检查Master组件服务状态。
- [uploadconfig]：更新配置
- [kubelet]：使用configMap配置kubelet。
- [patchnode]：更新CNI信息到Node上，通过注释的方式记录。
- [mark-control-plane]：为当前节点打标签，打了角色Master，和不可调度标签，这样默认就不会使用Master节点来运行Pod。
- [bootstrap-token]：生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到
- [addons]：安装附加组件CoreDNS和kube-proxy
说明：无论是初始化失败或者集群已经完全搭建成功，你都可以直接执行kubeadm reset命令清理集群或节点，然后重新执行kubeadm init或kubeadm join相关操作即可。
配置kubectl命令
无论在master节点或node节点，要能够执行kubectl命令必须进行以下配置：
root用户执行以下命令
cat << EOF >> ~/.bashrc export KUBECONFIG=/etc/kubernetes/admin.conf EOF source ~/.bashrc
普通用户执行以下命令（参考init时的输出结果）
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
等集群配置完成后，可以在所有master节点和node节点进行以上配置，以支持kubectl命令。针对node节点复制任意master节点/etc/kubernetes/admin.conf到本地。
查看当前状态
[root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 NotReady master 81s v1.14.1 [root@k8s-master01 ~]# kubectl -n kube-system get pod NAME READY STATUS RESTARTS AGE coredns-8686dcc4fd-cbrc5 0/1 Pending 0 64s coredns-8686dcc4fd-wqpwr 0/1 Pending 0 64s etcd-k8s-master01 1/1 Running 0 16s kube-apiserver-k8s-master01 1/1 Running 0 13s kube-controller-manager-k8s-master01 1/1 Running 0 25s kube-proxy-4vwbb 1/1 Running 0 65s kube-scheduler-k8s-master01 1/1 Running 0 4s [root@k8s-master01 ~]# kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health":"true"}
由于未安装网络插件，coredns处于pending状态，node处于notready状态。
安装网络插件
kubernetes支持多种网络方案，这里简单介绍常用的flannel和calico安装方法，选择其中一种方案进行部署即可。
以下操作在master01节点执行即可。
安装flannel网络插件：
由于kube-flannel.yml文件指定的镜像从coreos镜像仓库拉取，可能拉取失败，可以从dockerhub搜索相关镜像进行替换，另外可以看到yml文件中定义的网段地址段为10.244.0.0/16。
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml cat kube-flannel.yml | grep image cat kube-flannel.yml | grep 10.244 sed -i 's#quay.io/coreos/flannel:v0.11.0-amd64#willdockerhub/flannel:v0.11.0-amd64#g' kube-flannel.yml kubectl apply -f kube-flannel.yml
再次查看node和 Pod状态，全部为Running
[root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 Ready master 9m8s v1.14.1 [root@k8s-master01 ~]# kubectl -n kube-system get pod NAME READY STATUS RESTARTS AGE coredns-8686dcc4fd-cbrc5 1/1 Running 0 8m53s coredns-8686dcc4fd-wqpwr 1/1 Running 0 8m53s etcd-k8s-master01 1/1 Running 0 8m5s kube-apiserver-k8s-master01 1/1 Running 0 8m2s kube-controller-manager-k8s-master01 1/1 Running 0 8m14s kube-flannel-ds-amd64-vtppf 1/1 Running 0 115s kube-proxy-4vwbb 1/1 Running 0 8m54s kube-scheduler-k8s-master01 1/1 Running 0 7m53s
安装calico网络插件（可选）：
安装参考：https://docs.projectcalico.org/v3.6/getting-started/kubernetes/
kubectl apply -f https://docs.projectcalico.org/v3.6/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
注意该yaml文件中默认CIDR为192.168.0.0/16，需要与初始化时kube-config.yaml中的配置一致，如果不同请下载该yaml修改后运行。
加入master节点
从初始化输出或kubeadm-init.log中获取命令
kubeadm join 192.168.92.30:6444 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:c0a1021e5d63f509a0153724270985cdc22e46dc76e8e7b84d1fbb5e83566ea8 --experimental-control-plane --certificate-key 52f64a834454c3043fe7a0940f928611b6970205459fa19cb1193b33a288e7cc
依次将k8s-master02和k8s-master03加入到集群中，示例
[root@k8s-master02 ~]# kubeadm join 192.168.92.30:6444 --token abcdef.0123456789abcdef > --discovery-token-ca-cert-hash sha256:7b232b343577bd5fac312996b9fffb3c88f8f8bb39f46bf865ac9f9f52982b82 > --experimental-control-plane --certificate-key 11def25d624a2150b57715e21b0c393695bc6a70d932e472f75d24f747eb657e [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [preflight] Running pre-flight checks before initializing the new control plane instance [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master02 localhost] and IPs [192.168.92.11 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master02 localhost] and IPs [192.168.92.11 127.0.0.1 ::1] [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master02 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.92.11 192.168.92.30] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Valid certificates and keys now exist in "/etc/kubernetes/pki" [certs] Using the existing "sa" key [kubeconfig] Generating kubeconfig files [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [check-etcd] Checking that the etcd cluster is healthy [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Activating the kubelet service [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... [etcd] Announced new etcd member joining to the existing etcd cluster [etcd] Wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml" [etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s [upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [mark-control-plane] Marking the node k8s-master02 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node k8s-master02 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] This node has joined the cluster and a new control plane instance was created: * Certificate signing request was sent to apiserver and approval was received. * The Kubelet was informed of the new secure connection details. * Control plane (master) label and taint were applied to the new node. * The Kubernetes control plane instances scaled up. * A new etcd member was added to the local/stacked etcd cluster. To start administering your cluster from this node, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Run 'kubectl get nodes' to see this node join the cluster.
加入node节点
从kubeadm-init.log中获取命令
kubeadm join 192.168.92.30:6444 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:c0a1021e5d63f509a0153724270985cdc22e46dc76e8e7b84d1fbb5e83566ea8
示例
[root@k8s-node01 ~]# kubeadm join 192.168.92.30:6444 --token abcdef.0123456789abcdef > --discovery-token-ca-cert-hash sha256:7b232b343577bd5fac312996b9fffb3c88f8f8bb39f46bf865ac9f9f52982b82 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Activating the kubelet service [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
验证集群状态
查看nodes运行情况
[root@k8s-master01 ~]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-master01 Ready master 10h v1.14.1 192.168.92.10 <none> CentOS Linux 7 (Core) 3.10.0-957.10.1.el7.x86_64 docker://18.9.5 k8s-master02 Ready master 10h v1.14.1 192.168.92.11 <none> CentOS Linux 7 (Core) 3.10.0-957.10.1.el7.x86_64 docker://18.9.5 k8s-master03 Ready master 10h v1.14.1 192.168.92.12 <none> CentOS Linux 7 (Core) 3.10.0-957.10.1.el7.x86_64 docker://18.9.5 k8s-node01 Ready <none> 10h v1.14.1 192.168.92.13 <none> CentOS Linux 7 (Core) 3.10.0-957.10.1.el7.x86_64 docker://18.9.5
查看pod运行情况
[root@k8s-master03 ~]# kubectl -n kube-system get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-8686dcc4fd-6ttgv 1/1 Running 1 22m 10.244.2.3 k8s-master03 <none> <none> coredns-8686dcc4fd-dzvsx 1/1 Running 0 22m 10.244.3.3 k8s-node01 <none> <none> etcd-k8s-master01 1/1 Running 1 6m23s 192.168.92.10 k8s-master01 <none> <none> etcd-k8s-master02 1/1 Running 0 37m 192.168.92.11 k8s-master02 <none> <none> etcd-k8s-master03 1/1 Running 1 36m 192.168.92.12 k8s-master03 <none> <none> kube-apiserver-k8s-master01 1/1 Running 1 48m 192.168.92.10 k8s-master01 <none> <none> kube-apiserver-k8s-master02 1/1 Running 0 37m 192.168.92.11 k8s-master02 <none> <none> kube-apiserver-k8s-master03 1/1 Running 2 36m 192.168.92.12 k8s-master03 <none> <none> kube-controller-manager-k8s-master01 1/1 Running 2 48m 192.168.92.10 k8s-master01 <none> <none> kube-controller-manager-k8s-master02 1/1 Running 1 37m 192.168.92.11 k8s-master02 <none> <none> kube-controller-manager-k8s-master03 1/1 Running 1 35m 192.168.92.12 k8s-master03 <none> <none> kube-flannel-ds-amd64-d86ct 1/1 Running 0 37m 192.168.92.11 k8s-master02 <none> <none> kube-flannel-ds-amd64-l8clz 1/1 Running 0 36m 192.168.92.13 k8s-node01 <none> <none> kube-flannel-ds-amd64-vtppf 1/1 Running 1 42m 192.168.92.10 k8s-master01 <none> <none> kube-flannel-ds-amd64-zg4z5 1/1 Running 1 37m 192.168.92.12 k8s-master03 <none> <none> kube-proxy-4vwbb 1/1 Running 1 49m 192.168.92.10 k8s-master01 <none> <none> kube-proxy-gnk2v 1/1 Running 0 37m 192.168.92.11 k8s-master02 <none> <none> kube-proxy-kqm87 1/1 Running 0 36m 192.168.92.13 k8s-node01 <none> <none> kube-proxy-n5mdh 1/1 Running 2 37m 192.168.92.12 k8s-master03 <none> <none> kube-scheduler-k8s-master01 1/1 Running 2 48m 192.168.92.10 k8s-master01 <none> <none> kube-scheduler-k8s-master02 1/1 Running 1 37m 192.168.92.11 k8s-master02 <none> <none> kube-scheduler-k8s-master03 1/1 Running 2 36m 192.168.92.12 k8s-master03 <none> <none>
查看service
[root@k8s-master03 ~]# kubectl -n kube-system get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 51m
验证IPVS
查看kube-proxy日志，第一行输出Using ipvs Proxier.
[root@k8s-master01 ~]# kubectl -n kube-system logs -f kube-proxy-4vwbb I0426 16:05:03.156092 1 server_others.go:177] Using ipvs Proxier. W0426 16:05:03.156501 1 proxier.go:381] IPVS scheduler not specified, use rr by default I0426 16:05:03.156788 1 server.go:555] Version: v1.14.1 I0426 16:05:03.166269 1 conntrack.go:52] Setting nf_conntrack_max to 131072 I0426 16:05:03.169022 1 config.go:202] Starting service config controller I0426 16:05:03.169103 1 controller_utils.go:1027] Waiting for caches to sync for service config controller I0426 16:05:03.169182 1 config.go:102] Starting endpoints config controller I0426 16:05:03.169200 1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller I0426 16:05:03.269760 1 controller_utils.go:1034] Caches are synced for endpoints config controller I0426 16:05:03.270123 1 controller_utils.go:1034] Caches are synced for service config controller I0426 16:05:03.352400 1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/192.168.92.11:6443 I0426 16:05:03.352478 1 graceful_termination.go:174] Deleting rs: 10.96.0.1:443/TCP/192.168.92.11:6443
查看代理规则
[root@k8s-master01 ~]# ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.96.0.1:443 rr -> 192.168.92.10:6443 Masq 1 3 0 -> 192.168.92.11:6443 Masq 1 0 0 -> 192.168.92.12:6443 Masq 1 0 0 TCP 10.96.0.10:53 rr -> 10.244.0.5:53 Masq 1 0 0 -> 10.244.0.6:53 Masq 1 0 0 TCP 10.96.0.10:9153 rr -> 10.244.0.5:9153 Masq 1 0 0 -> 10.244.0.6:9153 Masq 1 0 0 UDP 10.96.0.10:53 rr -> 10.244.0.5:53 Masq 1 0 0 -> 10.244.0.6:53 Masq 1 0 0
etcd集群
执行以下命令查看etcd集群状态
kubectl -n kube-system exec etcd-k8s-master01 -- etcdctl --endpoints=https://192.168.92.10:2379 --ca-file=/etc/kubernetes/pki/etcd/ca.crt --cert-file=/etc/kubernetes/pki/etcd/server.crt --key-file=/etc/kubernetes/pki/etcd/server.key cluster-health
示例
[root@k8s-master01 ~]# kubectl -n kube-system exec etcd-k8s-master01 -- etcdctl > --endpoints=https://192.168.92.10:2379 > --ca-file=/etc/kubernetes/pki/etcd/ca.crt > --cert-file=/etc/kubernetes/pki/etcd/server.crt > --key-file=/etc/kubernetes/pki/etcd/server.key cluster-health member a94c223ced298a9 is healthy: got healthy result from https://192.168.92.12:2379 member 1db71d0384327b96 is healthy: got healthy result from https://192.168.92.11:2379 member e86955402ac20700 is healthy: got healthy result from https://192.168.92.10:2379 cluster is healthy
验证HA
在master01上执行关机操作，建议提前在其他节点配置kubectl命令支持。
[root@k8s-master01 ~]# shutdown -h now
在任意运行节点验证集群状态，master01节点NotReady，集群可正常访问：
[root@k8s-master02 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 NotReady master 19m v1.14.1 k8s-master02 Ready master 11m v1.14.1 k8s-master03 Ready master 10m v1.14.1 k8s-node01 Ready <none> 9m21s v1.14.1
查看网卡，vip自动漂移到master03节点
[root@k8s-master03 ~]# ip a |grep ens33 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.92.12/24 brd 192.168.92.255 scope global noprefixroute ens33 inet 192.168.92.30/24 scope global secondary ens33
原文链接：https://blog.csdn.net/networken/java/article/details/89599004