K8S集群二进制搭建3——部署Master Node

在k8s-master主机上操作，本篇文章主要部署三个组件——kube-apiserver，kube-controller-manager，kube-scheduler：

1.1、自建CA：

[root@k8s-master k8s]# pwd
/root/TLS/k8s

[root@k8s-master k8s]# vim ca-config.json

{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}

[root@k8s-master k8s]# vim ca-csr.json

{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
]
}

生成证书：

[root@k8s-master k8s]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

1.2、签发kube-apiserer HTTPS证书：

创建证书申请文件，hosts字段中的IP为所有Master、LB、VIP的IP地址，一个都不能少，为方便扩展可多预留几个IP：

[root@k8s-master k8s]# cat server-scr.json
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"192.168.1.206",
"192.168.1.207",
"192.168.1.208",
"192.168.1.209",
"192.168.1.210",
"192.168.1.211",
"192.168.1.213",
"192.168.1.213",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}

生成ssl证书：

[root@k8s-master k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

2.1 kuberneter安装：

到https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1183链接下下载kubernetes-server-linux-amd64.tar.gz软件包

[root@k8s-master opt]# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}

[root@k8s-master opt]# tar zxvf kubernetes-server-linux-amd64.tar.gz

[root@k8s-master etcd-v3.4.9-linux-amd64]# cd /opt/kubernetes/server/bin/

[root@k8s-master bin]# cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
[root@k8s-master bin]# cp kubectl /usr/bin/

2.1.1 部署kube-apiserver：

创建配置文件，每个字段作用：

• –logtostderr：启用日志

• —v：日志等级

• –log-dir：日志目录

• –etcd-servers：etcd集群地址

• –bind-address：监听地址

• –secure-port：https安全端口

• –advertise-address：集群通告地址

• –allow-privileged：启用授权

• –service-cluster-ip-range：Service虚拟IP地址段

• –enable-admission-plugins：准入控制模块

• –authorization-mode：认证授权，启用RBAC授权和节点自管理

• –enable-bootstrap-token-auth：启用TLS bootstrap机制

• –token-auth-file：bootstrap token文件

• –service-node-port-range：Service nodeport类型默认分配端口范围

• –kubelet-client-xxx：apiserver访问kubelet客户端证书

• –tls-xxx-file：apiserver https证书

• –etcd-xxxfile：连接Etcd集群证书

• –audit-log-xxx：审计日志

[root@k8s-master opt]# cat /opt/kubernetes/cfg/kube-apiserver.conf
KUBE_APISERVER_OPTS="--logtostderr=false
--v=2
--log-dir=/opt/kubernetes/logs
--etcd-servers=https://192.168.1.206:2379,https://192.168.1.207:2379,https://192.168.1.208:2379
--bind-address=192.168.1.206
--secure-port=6443
--advertise-address=192.168.1.206
--allow-privileged=true
--service-cluster-ip-range=10.0.0.0/24
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction
--authorization-mode=RBAC,Node
--enable-bootstrap-token-auth=true
--token-auth-file=/opt/kubernetes/cfg/token.csv
--service-node-port-range=30000-32767
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem
--tls-cert-file=/opt/kubernetes/ssl/server.pem
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem
--client-ca-file=/opt/kubernetes/ssl/ca.pem
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem
--etcd-cafile=/opt/etcd/ssl/ca.pem
--etcd-certfile=/opt/etcd/ssl/server.pem
--etcd-keyfile=/opt/etcd/ssl/server-key.pem
--audit-log-maxage=30
--audit-log-maxbackup=3
--audit-log-maxsize=100
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

拷贝证书到配置文件中对应位置：

[root@k8s-master opt]# cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/

2.1.2 TLS Bootstrapping机制引入：

      Master apiserver启用TLS Bootstrapping认证后，Node节点kubelet和kube-proxy要与kube-apiserver进行通信，必须要使用C签发的有效证书才行，当Node节点过多，证书颁发需要大量工作，会增加集群扩展的复杂度。为了简化流程，kubernetes引入TLS Bootstrapping机制自动颁发证书，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署，所以当集群规模较大是建议在客户端使用该方式签发。工作流程：

 配置tocken文件（tocken，用户名，UID，用户组）：

tocken生成：[root@k8s-master logs]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '

[root@k8s-master opt]# cat /opt/kubernetes/cfg/token.csv
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"

设置systemd启动：

[root@k8s-master opt]# cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target

[root@k8s-master opt]# systemctl daemon-reload
[root@k8s-master opt]# systemctl start kube-apiserver.service
[root@k8s-master opt]# systemctl enable kube-apiserver.service

排错，服务器不能正常启动，开两个窗口（先执行以下命令，在另一个窗口重启systemctl restart kube-apiserver.service ）：

[root@k8s-master bin]# journalctl -f -u kube-apiserver.service

 通过查看信息可以发现，配置文件中的参数全部被忽略掉了，缺少一些必备的参数，导致服务服务启动，修改配置文件，每个参数之间不能换行：

KUBE_APISERVER_OPTS="--logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --etcd-servers=https://192.168.1.206:2379,https://192.168.1.207:2379,https://192.168.1.208:2379 --bind-address=192.168.1.206 --secure-port=6443 --advertise-address=192.168.1.206 --allow-privileged=true --service-cluster-ip-range=10.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --enable-bootstrap-token-auth=true --token-auth-file=/opt/kubernetes/cfg/token.csv --service-node-port-range=30000-32767 --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem --tls-cert-file=/opt/kubernetes/ssl/server.pem  --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem --client-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --etcd-cafile=/opt/etcd/ssl/ca.pem --etcd-certfile=/opt/etcd/ssl/server.pem --etcd-keyfile=/opt/etcd/ssl/server-key.pem --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

然后重启服务即可：

3.1 部署kube-controller-manager

编辑配置文件：

• –master：通过本地非安全本地端口8080连接apiserver。

• –leader-elect：当该组件启动多个时，自动选举（HA）

• –cluster-signing-cert-file/–cluster-signing-key-file：自动为kubelet颁发证书的CA，与apiserver保持一致

[root@k8s-master ~]# cat /opt/kubernetes/cfg/kube-controller-manager.conf

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-elect=true --master=127.0.0.1:8080 --bind-address=127.0.0.1 --allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16 --service-cluster-ip-range=10.0.0.0/24 --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  --root-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem --experimental-cluster-signing-duration=87600h0m0s"

配置systemctl服务：

[root@k8s-master ~]# cat /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target

[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl start kube-controller-manager.service
[root@k8s-master ~]# systemctl enable kube-controller-manager.service

4.1部署kube-scheduler

• –master：通过本地非安全本地端口8080连接apiserver。

• –leader-elect：当该组件启动多个时，自动选举（HA）

[root@k8s-master ~]# cat /opt/kubernetes/cfg/kube-scheduler.conf
KUBE_SCHEDULER_OPTS="--logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-elect --master=127.0.0.1:8080 --bind-address=127.0.0.1"

配置systemctl服务：

[root@k8s-master ~]# cat /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target

[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl start kube-scheduler
[root@k8s-master ~]# systemctl enable kube-scheduler

查看集群状态：