我们这里接着之前Kubernetes容器集群部署（二进制方式）上（https://www.cnblogs.com/qingbaizhinian/p/14290333.html）继续往下部署：
8.部署master节点前期基础组件环境准备
master节点的kube-apiserver、kube-scheduler 和 kube-controller-manager 均以多实例模式运行：kube-scheduler 和 kube-controller-manager 会自动选举产生一个 leader 实例，其它实例处于阻塞模式，当 leader 挂了后，重新选举产生新的 leader，从而保证服务可用性；kube-apiserver 是无状态的，需要通过 kube-nginx 进行代理访问，从而保证服务可用性；下面部署命令均在k8s-master01节点上执行，然后远程分发文件和执行命令。
1.下载最新的二进制文件 这里我们在上篇部署文档，部署kubectl命令行工具下载过相关二进制文件包了，并进行了解压了。我们现在回顾一下命令，此时我们真正要执行的命令是下面黄底纹的命令 [root@k8s-master01 ~]# cd /opt/k8s/work [root@k8s-master01 work]# wget https://storage.googleapis.com/kubernetes-release/release/v1.19.6/kubernetes-server-linux-amd64.tar.gz [root@k8s-master01 work]# tar -zxvf kubernetes-server-linux-amd64.tar.gz [root@k8s-master01 work]# cd kubernetes [root@k8s-master01 kubernetes]# tar -zxvf kubernetes-src.tar.gz 2.将二进制文件拷贝到所有 master 节点： [root@k8s-master01 ~]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" scp kubernetes/server/bin/{apiextensions-apiserver,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubectl,kubelet,mounter} root@${node_master_ip}:/opt/k8s/bin/ ssh root@${node_master_ip} "chmod +x /opt/k8s/bin/*" done
9.部署高可用 kube-apiserver 集群
这里部署一个三实例kube-apiserver集群环境，它们通过nginx四层代理进行访问，对外提供一个统一的vip地址，从而保证服务可用性。下面部署命令均在k8s-master01节点上执行，然后远程分发文件和执行命令。
1.创建kubernetes 证书和私钥
1.创建证书签名请求： [root@k8s-master01 ~]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# cat > kubernetes-csr.json <<EOF { "CN": "kubernetes", "hosts": [ "127.0.0.1", "172.31.46.47", "172.31.46.28", "172.31.46.63", "172.31.46.67", "${CLUSTER_KUBERNETES_SVC_IP}", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Hefei", "L": "Hefei", "O": "k8s", "OU": "4Paradigm" } ] } EOF 解释说明： • hosts 字段指定授权使用该证书的 IP 或域名列表，这里列出了 VIP 、apiserver 节点 IP、kubernetes 服务 IP 和域名； • 域名最后字符不能是 .(如不能为 kubernetes.default.svc.cluster.local.)，否则解析时失败，提示： x509: cannot parse dnsName "kubernetes.default.svc.cluster.local."； • 如果使用非 cluster.local 域名，如 opsnull.com，则需要修改域名列表中的最后两个域名为：kubernetes.default.svc.opsnull、kubernetes.default.svc.opsnull.com • kubernetes 服务 IP 是 apiserver 自动创建的，一般是 --service-cluster-ip-range 参数指定的网段的第一个IP，后续可以通过如下命令获取： [root@k8s-master01 work]# kubectl get svc kubernetes The connection to the server 172.16.60.250:8443 was refused - did you specify the right host or port? 上面报错是因为kube-apiserver服务此时没有启动，后续待apiserver服务启动后，以上命令就可以获得了。 2.生成证书和私钥： [root@k8s-master01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes [root@k8s-master01 work]# ls kubernetes*pem kubernetes-key.pem kubernetes.pem 3.将生成的证书和私钥文件拷贝到所有 master 节点： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" ssh root@${node_master_ip} "mkdir -p /etc/kubernetes/cert" scp kubernetes*.pem root@${node_master_ip}:/etc/kubernetes/cert/ done
2.创建加密配置文件
[root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# cat > encryption-config.yaml <<EOF kind: EncryptionConfig apiVersion: v1 resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: ${ENCRYPTION_KEY} - identity: {} EOF 将加密配置文件拷贝到 master 节点的 /etc/kubernetes 目录下： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" scp encryption-config.yaml root@${node_master_ip}:/etc/kubernetes/ done
3.创建审计策略文件
[root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# cat > audit-policy.yaml <<EOF apiVersion: audit.k8s.io/v1beta1 kind: Policy rules: # The following requests were manually identified as high-volume and low-risk, so drop them. - level: None resources: - group: "" resources: - endpoints - services - services/status users: - 'system:kube-proxy' verbs: - watch - level: None resources: - group: "" resources: - nodes - nodes/status userGroups: - 'system:nodes' verbs: - get - level: None namespaces: - kube-system resources: - group: "" resources: - endpoints users: - 'system:kube-controller-manager' - 'system:kube-scheduler' - 'system:serviceaccount:kube-system:endpoint-controller' verbs: - get - update - level: None resources: - group: "" resources: - namespaces - namespaces/status - namespaces/finalize users: - 'system:apiserver' verbs: - get # Don't log HPA fetching metrics. - level: None resources: - group: metrics.k8s.io users: - 'system:kube-controller-manager' verbs: - get - list # Don't log these read-only URLs. - level: None nonResourceURLs: - '/healthz*' - /version - '/swagger*' # Don't log events requests. - level: None resources: - group: "" resources: - events # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes - level: Request omitStages: - RequestReceived resources: - group: "" resources: - nodes/status - pods/status users: - kubelet - 'system:node-problem-detector' - 'system:serviceaccount:kube-system:node-problem-detector' verbs: - update - patch - level: Request omitStages: - RequestReceived resources: - group: "" resources: - nodes/status - pods/status userGroups: - 'system:nodes' verbs: - update - patch # deletecollection calls can be large, don't log responses for expected namespace deletions - level: Request omitStages: - RequestReceived users: - 'system:serviceaccount:kube-system:namespace-controller' verbs: - deletecollection # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data, # so only log at the Metadata level. - level: Metadata omitStages: - RequestReceived resources: - group: "" resources: - secrets - configmaps - group: authentication.k8s.io resources: - tokenreviews # Get repsonses can be large; skip them. - level: Request omitStages: - RequestReceived resources: - group: "" - group: admissionregistration.k8s.io - group: apiextensions.k8s.io - group: apiregistration.k8s.io - group: apps - group: authentication.k8s.io - group: authorization.k8s.io - group: autoscaling - group: batch - group: certificates.k8s.io - group: extensions - group: metrics.k8s.io - group: networking.k8s.io - group: policy - group: rbac.authorization.k8s.io - group: scheduling.k8s.io - group: settings.k8s.io - group: storage.k8s.io verbs: - get - list - watch # Default level for known APIs - level: RequestResponse omitStages: - RequestReceived resources: - group: "" - group: admissionregistration.k8s.io - group: apiextensions.k8s.io - group: apiregistration.k8s.io - group: apps - group: authentication.k8s.io - group: authorization.k8s.io - group: autoscaling - group: batch - group: certificates.k8s.io - group: extensions - group: metrics.k8s.io - group: networking.k8s.io - group: policy - group: rbac.authorization.k8s.io - group: scheduling.k8s.io - group: settings.k8s.io - group: storage.k8s.io # Default level for all other requests. - level: Metadata omitStages: - RequestReceived EOF 分发审计策略文件： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" scp audit-policy.yaml root@${node_master_ip}:/etc/kubernetes/audit-policy.yaml done
4.创建后续访问 metrics-server 使用的证书
1.创建证书签名请求: [root@k8s-master01 work]# cat > proxy-client-csr.json <<EOF { "CN": "aggregator", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Hefei", "L": "Hefei", "O": "k8s", "OU": "4Paradigm" } ] } EOF CN 名称为 aggregator，需要与 metrics-server 的 --requestheader-allowed-names 参数配置一致，否则访问会被 metrics-server 拒绝； 2.生成证书和私钥： [root@k8s-master01 work]# cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client [root@k8s-master01 work]# ls proxy-client*.pem proxy-client-key.pem proxy-client.pem 3.将生成的证书和私钥文件拷贝到所有 master 节点： [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" scp proxy-client*.pem root@${node_master_ip}:/etc/kubernetes/cert/ done
5. 创建 kube-apiserver systemd unit 模板文件
[root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# cat > kube-apiserver.service.template <<EOF [Unit] Description=Kubernetes API Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] WorkingDirectory=${K8S_DIR}/kube-apiserver ExecStart=/opt/k8s/bin/kube-apiserver \ --advertise-address=##NODE_MASTER_IP## \ --default-not-ready-toleration-seconds=360 \ --default-unreachable-toleration-seconds=360 \--max-mutating-requests-inflight=2000 \ --max-requests-inflight=4000 \ --default-watch-cache-size=200 \ --delete-collection-workers=2 \ --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \ --etcd-cafile=/etc/kubernetes/cert/ca.pem \ --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \ --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \ --etcd-servers=${ETCD_ENDPOINTS} \ --bind-address=##NODE_MASTER_IP## \ --secure-port=6443 \ --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \ --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \ --insecure-port=0 \--audit-log-maxage=15 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-mode=batch \ --audit-log-truncate-enabled \ --audit-log-batch-buffer-size=20000 \ --audit-log-batch-max-size=2 \ --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \ --audit-policy-file=/etc/kubernetes/audit-policy.yaml \ --profiling \ --anonymous-auth=false \ --client-ca-file=/etc/kubernetes/cert/ca.pem \ --enable-bootstrap-token-auth \ --requestheader-allowed-names="aggregator" \ --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \ --requestheader-extra-headers-prefix="X-Remote-Extra-" \ --requestheader-group-headers=X-Remote-Group \ --requestheader-username-headers=X-Remote-User \ --service-account-key-file=/etc/kubernetes/cert/ca.pem \ --authorization-mode=Node,RBAC \ --runtime-config=api/all=true \ --enable-admission-plugins=NodeRestriction \ --allow-privileged=true \ --apiserver-count=3 \ --event-ttl=168h \ --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \ --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \ --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \ --kubelet-https=true \ --kubelet-timeout=10s \ --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \ --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \ --service-cluster-ip-range=${SERVICE_CIDR} \ --service-node-port-range=${NODE_PORT_RANGE} \ --logtostderr=true \ --enable-aggregator-routing=true \ --v=2 Restart=on-failure RestartSec=10 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF 解释说明： --advertise-address：apiserver 对外通告的 IP（kubernetes 服务后端节点 IP）； --default-*-toleration-seconds：设置节点异常相关的阈值； --max-*-requests-inflight：请求相关的最大阈值； --etcd-*：访问 etcd 的证书和 etcd 服务器地址； --experimental-encryption-provider-config：指定用于加密 etcd 中 secret 的配置； --bind-address： https 监听的 IP，不能为 127.0.0.1，否则外界不能访问它的安全端口 6443； --secret-port：https 监听端口； --insecure-port=0：关闭监听 http 非安全端口(8080)； --tls-*-file：指定 apiserver 使用的证书、私钥和 CA 文件； --audit-*：配置审计策略和审计日志文件相关的参数； --client-ca-file：验证 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)请求所带的证书； --enable-bootstrap-token-auth：启用 kubelet bootstrap 的 token 认证； --requestheader-*：kube-apiserver 的 aggregator layer 相关的配置参数，proxy-client & HPA 需要使用； --requestheader-client-ca-file：用于签名 --proxy-client-cert-file 和 --proxy-client-key-file 指定的证书；在启用了 metric aggregator 时使用； 如果 --requestheader-allowed-names 不为空，则--proxy-client-cert-file 证书的 CN 必须位于 allowed-names 中，默认为 aggregator; --service-account-key-file：签名 ServiceAccount Token 的公钥文件，kube-controller-manager 的 --service-account-private-key-file 指定私钥文件，两者配对使用； --runtime-config=api/all=true： 启用所有版本的 APIs，如 autoscaling/v2alpha1； --authorization-mode=Node,RBAC、--anonymous-auth=false： 开启 Node 和 RBAC 授权模式，拒绝未授权的请求； --enable-admission-plugins：启用一些默认关闭的 plugins； --allow-privileged：运行执行 privileged 权限的容器； --apiserver-count=3：指定 apiserver 实例的数量； --event-ttl：指定 events 的保存时间； --kubelet-*：如果指定，则使用 https 访问 kubelet APIs；需要为证书对应的用户(上面 kubernetes*.pem 证书的用户为 kubernetes) 用户定义 RBAC 规则，否则访问 kubelet API 时提示未授权； --proxy-client-*：apiserver 访问 metrics-server 使用的证书； --service-cluster-ip-range： 指定 Service Cluster IP 地址段； --service-node-port-range： 指定 NodePort 的端口范围； 注意： 如果kube-apiserver机器没有运行 kube-proxy，则需要添加 --enable-aggregator-routing=true 参数（这里master节点没有作为node节点使用，故没有运行kube-proxy，需要加这个参数） requestheader-client-ca-file 指定的 CA 证书，必须具有 client auth and server auth！！ 为各节点创建和分发 kube-apiserver systemd unit 文件 替换模板文件中的变量，为各节点生成 systemd unit 文件： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for (( i=0; i < 3; i++ )) do sed -e "s/##NODE_MASTER_NAME##/${NODE_MASTER_NAMES[i]}/" -e "s/##NODE_MASTER_IP##/${NODE_MASTER_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-${NODE_MASTER_IPS[i]}.service done 其中：NODE_NAMES 和 NODE_IPS 为相同长度的 bash 数组，分别为节点名称和对应的 IP； [root@k8s-master01 work]# ll kube-apiserver*.service -rw-r--r-- 1 root root 2719 Feb 9 16:42 kube-apiserver-172.31.46.28.service -rw-r--r-- 1 root root 2719 Feb 9 16:42 kube-apiserver-172.31.46.63.service -rw-r--r-- 1 root root 2719 Feb 9 16:42 kube-apiserver-172.31.46.67.service 分发生成的 systemd unit 文件, 文件重命名为 kube-apiserver.service; [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" scp kube-apiserver-${node_master_ip}.service root@${node_master_ip}:/etc/systemd/system/kube-apiserver.service done
6.启动kube-apiserver 服务
[root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" ssh root@${node_master_ip} "mkdir -p ${K8S_DIR}/kube-apiserver" ssh root@${node_master_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver" done 注意：启动服务前必须先创建工作目录； 检查 kube-apiserver 运行状态 [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" ssh root@${node_master_ip} "systemctl status kube-apiserver |grep 'Active:'" done 预期输出：
>>> 172.31.46.28
Active: active (running) since Wed 2021-02-10 20:49:43 CST; 22s ago
>>> 172.31.46.63
Active: active (running) since Wed 2021-02-10 20:49:47 CST; 17s ago
>>> 172.31.46.67
Active: active (running) since Wed 2021-02-10 20:49:52 CST; 13s ago
确保状态为 active (running)，否则查看日志，确认原因（journalctl -u kube-apiserver）7.打印 kube-apiserver 写入 etcd 的数据
[root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# ETCDCTL_API=3 etcdctl --endpoints=${ETCD_ENDPOINTS} --cacert=/opt/k8s/work/ca.pem --cert=/opt/k8s/work/etcd.pem --key=/opt/k8s/work/etcd-key.pem get /registry/ --prefix --keys-only 预期会打印出很多写入到etcd中的数据信息
8.检查集群信息
[root@k8s-master01 work]# kubectl cluster-info Kubernetes master is running at https://172.31.46.47:8443 To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. [root@k8s-master01 work]# kubectl get all --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 16m 查看集群状态信息 [root@k8s-master01 work]# kubectl get componentstatuses Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused controller-manager Unhealthy Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused etcd-0 Healthy {"health":"true"} etcd-1 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"} controller-managerhe 和 schedule状态为Unhealthy，是因为此时还没有部署这两个组件，待后续部署好之后再查看~ 这里注意： -> 如果执行 kubectl 命令式时输出如下错误信息，则说明使用的 ~/.kube/config 文件不对，请切换到正确的账户后再执行该命令： The connection to the server localhost:8080 was refused - did you specify the right host or port? -> 执行 kubectl get componentstatuses 命令时，apiserver 默认向 127.0.0.1 发送请求。当 controller-manager、scheduler 以集群模式运行时，有可能和kube-apiserver 不在一台机器上，这时 controller-manager 或 scheduler 的状态为 Unhealthy，但实际上它们工作正常。
9.检查 kube-apiserver 监听的端口
[root@k8s-master01 work]# netstat -lnpt|grep kube tcp 0 0 172.31.46.28:6443 0.0.0.0:* LISTEN 3106044/kube-apiser 需要注意： 6443: 接收 https 请求的安全端口，对所有请求做认证和授权； 由于关闭了非安全端口，故没有监听 8080；
10.授予 kube-apiserver 访问 kubelet API 的权限
在执行 kubectl exec、run、logs 等命令时，apiserver 会将请求转发到 kubelet 的 https 端口。
这里定义 RBAC 规则，授权 apiserver 使用的证书（kubernetes.pem）用户名（CN：kuberntes）访问 kubelet API 的权限：
[root@k8s-master01 work]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes
11.查看kube-apiserver输出的metrics
需要用到根证书 使用nginx的代理端口获取metrics [root@k8s-master01 work]# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.31.46.47:8443/metrics|head # HELP aggregator_openapi_v2_regeneration_count [ALPHA] Counter of OpenAPI v2 spec regeneration count broken down by causing APIService name and reason. # TYPE aggregator_openapi_v2_regeneration_count counter aggregator_openapi_v2_regeneration_count{apiservice="*",reason="startup"} 0 # HELP aggregator_openapi_v2_regeneration_duration [ALPHA] Gauge of OpenAPI v2 spec regeneration duration in seconds. # TYPE aggregator_openapi_v2_regeneration_duration gauge aggregator_openapi_v2_regeneration_duration{reason="startup"} 0.646590329 # HELP aggregator_unavailable_apiservice [ALPHA] Gauge of APIServices which are marked as unavailable broken down by APIService name. # TYPE aggregator_unavailable_apiservice gauge aggregator_unavailable_apiservice{name="v1."} 0 aggregator_unavailable_apiservice{name="v1.admissionregistration.k8s.io"} 0 直接使用kube-apiserver节点端口获取metrics [root@k8s-master01 work]# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.31.46.28:6443/metrics|head [root@k8s-master01 work]# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.31.46.63:6443/metrics|head [root@k8s-master01 work]# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.31.46.67:6443/metrics|head
10.部署高可用 kube-controller-manager 集群
该集群包含 3 个节点，启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用时，阻塞的节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。为保证通信安全，本文档先生成 x509 证书和私钥，kube-controller-manager 在如下两种情况下使用该证书：与 kube-apiserver 的安全端口通信; 在安全端口(https，10252) 输出 prometheus 格式的 metrics；下面部署命令均在k8s-master01节点上执行，然后远程分发文件和执行命令。
1.创建 kube-controller-manager 证书和私钥
1.创建证书签名请求： [root@k8s-master01 ~]# cd /opt/k8s/work [root@k8s-master01 work]# cat > kube-controller-manager-csr.json <<EOF { "CN": "system:kube-controller-manager", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "172.31.46.28", "172.31.46.63", "172.31.46.67" ], "names": [ { "C": "CN", "ST": "Hefei", "L": "Hefei", "O": "system:kube-controller-manager", "OU": "4Paradigm" } ] } EOF • hosts 列表包含所有 kube-controller-manager 节点 IP； • CN 为 system:kube-controller-manager、O 为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限。 2.生成证书和私钥 [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager [root@k8s-master01 work]# ll kube-controller-manager*pem -rw------- 1 root root 1675 Feb 11 09:20 kube-controller-manager-key.pem -rw-r--r-- 1 root root 1505 Feb 11 09:20 kube-controller-manager.pem 3.将生成的证书和私钥分发到所有 master 节点： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" scp kube-controller-manager*.pem root@${node_master_ip}:/etc/kubernetes/cert/ done
2.创建和分发kubeconfig 文件
kube-controller-manager 使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-controller-manager 证书： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# kubectl config set-cluster kubernetes --certificate-authority=/opt/k8s/work/ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=kube-controller-manager.kubeconfig [root@k8s-master01 work]# kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig [root@k8s-master01 work]# kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig [root@k8s-master01 work]# kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig 分发 kubeconfig 到所有 master 节点： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" scp kube-controller-manager.kubeconfig root@${node_master_ip}:/etc/kubernetes/ done
3.创建和分发kube-controller-manager system unit 文件
[root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# cat > kube-controller-manager.service.template <<EOF [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] WorkingDirectory=${K8S_DIR}/kube-controller-manager ExecStart=/opt/k8s/bin/kube-controller-manager \ --profiling \ --cluster-name=kubernetes \ --controllers=*,bootstrapsigner,tokencleaner \ --kube-api-qps=1000 \ --kube-api-burst=2000 \ --leader-elect \ --use-service-account-credentials=true \ --concurrent-service-syncs=2 \ --bind-address=0.0.0.0 \
--secure-port=10252 \ --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \ --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \
--port=0 \ --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \ --client-ca-file=/etc/kubernetes/cert/ca.pem \ --requestheader-allowed-names="aggregator" \ --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \ --requestheader-extra-headers-prefix="X-Remote-Extra-" \ --requestheader-group-headers=X-Remote-Group \ --requestheader-username-headers=X-Remote-User \ --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \ --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \ --experimental-cluster-signing-duration=8760h \ --horizontal-pod-autoscaler-sync-period=10s \ --concurrent-deployment-syncs=10 \ --concurrent-gc-syncs=30 \ --node-cidr-mask-size=24 \ --service-cluster-ip-range=${SERVICE_CIDR} \ --pod-eviction-timeout=6m \ --terminated-pod-gc-threshold=10000 \ --root-ca-file=/etc/kubernetes/cert/ca.pem \ --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \ --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \ --logtostderr=true \ --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF 解释说明： 下面两行一般要去掉，否则执行"kubectl get cs"检查集群状态时，controller-manager状态会为"Unhealthy" --port=0：关闭监听非安全端口（http），同时 --address 参数无效，--bind-address 参数有效； --secure-port=10252 --bind-address=0.0.0.0: 在所有网络接口监听 10252 端口的 https /metrics 请求； --kubeconfig：指定 kubeconfig 文件路径，kube-controller-manager 使用它连接和验证 kube-apiserver； --authentication-kubeconfig 和 --authorization-kubeconfig：kube-controller-manager 使用它连接 apiserver，对 client 的请求进行认证和授权。kube-controller-manager 不再使用 --tls-ca-file 对请求 https metrics 的 Client 证书进行校验。如果没有配置这两个 kubeconfig 参数，则 client 连接 kube-controller-manager https 端口的请求会被拒绝(提示权限不足)。 --cluster-signing-*-file：签名 TLS Bootstrap 创建的证书； --experimental-cluster-signing-duration：指定 TLS Bootstrap 证书的有效期； --root-ca-file：放置到容器 ServiceAccount 中的 CA 证书，用来对 kube-apiserver 的证书进行校验； --service-account-private-key-file：签名 ServiceAccount 中 Token 的私钥文件，必须和 kube-apiserver 的 --service-account-key-file 指定的公钥文件配对使用； --service-cluster-ip-range ：指定 Service Cluster IP 网段，必须和 kube-apiserver 中的同名参数一致； --leader-elect=true：集群运行模式，启用选举功能；被选为 leader 的节点负责处理工作，其它节点为阻塞状态； --controllers=*,bootstrapsigner,tokencleaner：启用的控制器列表，tokencleaner 用于自动清理过期的 Bootstrap token； --horizontal-pod-autoscaler-*：custom metrics 相关参数，支持 autoscaling/v2alpha1； --tls-cert-file、--tls-private-key-file：使用 https 输出 metrics 时使用的 Server 证书和秘钥； --use-service-account-credentials=true: kube-controller-manager 中各 controller 使用 serviceaccount 访问 kube-apiserver； 1.为各节点创建和分发 kube-controller-mananger systemd unit 文件 替换模板文件中的变量，为各节点创建 systemd unit 文件： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for (( i=0; i < 3; i++ )) do sed -e "s/##NODE_MASTER_NAME##/${NODE_MASTER_NAMES[i]}/" -e "s/##NODE_MASTER_IP##/${NODE_MASTER_IPS[i]}/" kube-controller-manager.service.template > kube-controller-manager-${NODE_MASTER_IPS[i]}.service done 注意： NODE_NAMES 和 NODE_IPS 为相同长度的 bash 数组，分别为节点名称和对应的 IP； [root@k8s-master01 work]# ll kube-controller-manager*.service -rw-r--r-- 1 root root 1876 Feb 11 12:56 kube-controller-manager-172.31.46.28.service -rw-r--r-- 1 root root 1839 Feb 11 09:45 kube-controller-manager-172.31.46.63.service -rw-r--r-- 1 root root 1839 Feb 11 09:45 kube-controller-manager-172.31.46.67.service 2.分发到所有 master 节点： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" scp kube-controller-manager-${node_master_ip}.service root@${node_master_ip}:/etc/systemd/system/kube-controller-manager.service done 注意：文件重命名为 kube-controller-manager.service; 3.启动 kube-controller-manager 服务 [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" ssh root@${node_master_ip} "mkdir -p ${K8S_DIR}/kube-controller-manager" ssh root@${node_master_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager" done 注意：启动服务前必须先创建工作目录； 4.检查服务运行状态 预期输出结果： >>> 172.31.46.28 Active: active (running) since Sun 2021-02-14 21:25:58 CST; 36s ago >>> 172.31.46.63 Active: active (running) since Sun 2021-02-14 21:25:58 CST; 36s ago >>> 172.31.46.67 Active: active (running) since Sun 2021-02-14 21:25:58 CST; 36s ago 确保状态为 active (running)，否则查看日志，确认原因（journalctl -u kube-controller-manager） kube-con[root@k8s-master01 work]# netstat -lnpt |grep kube-control tcp6 0 0 :::10252 :::* LISTEN 951896/kube-control troller-manager 监听 10252 端口，接收 https 请求： 检查集群状态，controller-manager的状态为"ok" 注意：当kube-controller-manager集群中的1个或2个节点的controller-manager服务挂掉，只要有一个节点的controller-manager服务活着， 则集群中controller-manager的状态仍然为"ok",仍然会继续提供服务！ [root@k8s-master01 work]# kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused controller-manager Healthy ok etcd-2 Healthy {"health":"true"} etcd-0 Healthy {"health":"true"} etcd-1 Healthy {"health":"true"}
4.查看输出的 metrics
注意：以下命令在3台kube-controller-manager节点上执行。 由于在kube-controller-manager启动文件中关掉了"--port=0"和"--secure-port=10252"这两个参数，则只能通过http方式获取到kube-controller-manager 输出的metrics信息。kube-controller-manager一般不会被访问，只有在监控时采集metrcis指标数据时被访问。 [root@k8s-master01 work]# curl -s http://172.31.46.28:10252/metrics|head # HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend. # TYPE apiserver_audit_event_total counter apiserver_audit_event_total 0 # HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend. # TYPE apiserver_audit_requests_rejected_total counter apiserver_audit_requests_rejected_total 0 # HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request. # TYPE apiserver_client_certificate_expiration_seconds histogram apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0 [root@k8s-master01 work]# curl -s --cacert /etc/kubernetes/cert/ca.pem http://172.31.46.28:10252/metrics |head # HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend. # TYPE apiserver_audit_event_total counter apiserver_audit_event_total 0 # HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend. # TYPE apiserver_audit_requests_rejected_total counter apiserver_audit_requests_rejected_total 0 # HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request. # TYPE apiserver_client_certificate_expiration_seconds histogram apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0 [root@k8s-master01 work]# curl -s --cacert /etc/kubernetes/cert/ca.pem http://127.0.0.1:10252/metrics |head # HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend. # TYPE apiserver_audit_event_total counter apiserver_audit_event_total 0 # HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend. # TYPE apiserver_audit_requests_rejected_total counter apiserver_audit_requests_rejected_total 0 # HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request. # TYPE apiserver_client_certificate_expiration_seconds histogram apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0 [root@k8s-master01 work]# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem http://172.31.46.28:10252/metrics |head # HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend. # TYPE apiserver_audit_event_total counter apiserver_audit_event_total 0 # HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend. # TYPE apiserver_audit_requests_rejected_total counter apiserver_audit_requests_rejected_total 0 # HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request. # TYPE apiserver_client_certificate_expiration_seconds histogram apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0
5.kube-controller-manager 的权限
ClusteRole system:kube-controller-manager 的权限很小，只能创建 secret、serviceaccount 等资源对象，各 controller 的权限分散到 ClusterRole system:controller:XXX 中： [root@k8s-master01 work]# kubectl describe clusterrole system:kube-controller-manager Name: system:kube-controller-manager Labels: kubernetes.io/bootstrapping=rbac-defaults Annotations: rbac.authorization.kubernetes.io/autoupdate: true PolicyRule: Resources Non-Resource URLs Resource Names Verbs --------- ----------------- -------------- ----- secrets [] [] [create delete get update] serviceaccounts [] [] [create get update] events [] [] [create patch update] events.events.k8s.io [] [] [create patch update] endpoints [] [] [create] serviceaccounts/token [] [] [create] tokenreviews.authentication.k8s.io [] [] [create] subjectaccessreviews.authorization.k8s.io [] [] [create] leases.coordination.k8s.io [] [] [create] endpoints [] [kube-controller-manager] [get update] leases.coordination.k8s.io [] [kube-controller-manager] [get update] configmaps [] [] [get] namespaces [] [] [get] *.* [] [] [list watch] 需要在 kube-controller-manager 的启动参数中添加 --use-service-account-credentials=true 参数，这样 main controller 会为各 controller 创建对应的 ServiceAccount XXX-controller。 内置的 ClusterRoleBinding system:controller:XXX 将赋予各 XXX-controller ServiceAccount 对应的 ClusterRole system:controller:XXX 权限。 [root@k8s-master01 work]# kubectl get clusterrole|grep controller system:controller:attachdetach-controller 2021-02-10T12:44:44Z system:controller:certificate-controller 2021-02-10T12:44:44Z system:controller:clusterrole-aggregation-controller 2021-02-10T12:44:44Z system:controller:cronjob-controller 2021-02-10T12:44:44Z system:controller:daemon-set-controller 2021-02-10T12:44:44Z system:controller:deployment-controller 2021-02-10T12:44:44Z system:controller:disruption-controller 2021-02-10T12:44:44Z system:controller:endpoint-controller 2021-02-10T12:44:44Z system:controller:endpointslice-controller 2021-02-10T12:44:44Z system:controller:endpointslicemirroring-controller 2021-02-10T12:44:44Z system:controller:expand-controller 2021-02-10T12:44:44Z system:controller:generic-garbage-collector 2021-02-10T12:44:44Z system:controller:horizontal-pod-autoscaler 2021-02-10T12:44:44Z system:controller:job-controller 2021-02-10T12:44:44Z system:controller:namespace-controller 2021-02-10T12:44:44Z system:controller:node-controller 2021-02-10T12:44:44Z system:controller:persistent-volume-binder 2021-02-10T12:44:44Z system:controller:pod-garbage-collector 2021-02-10T12:44:44Z system:controller:pv-protection-controller 2021-02-10T12:44:44Z system:controller:pvc-protection-controller 2021-02-10T12:44:44Z system:controller:replicaset-controller 2021-02-10T12:44:44Z system:controller:replication-controller 2021-02-10T12:44:44Z system:controller:resourcequota-controller 2021-02-10T12:44:44Z system:controller:route-controller 2021-02-10T12:44:44Z system:controller:service-account-controller 2021-02-10T12:44:44Z system:controller:service-controller 2021-02-10T12:44:44Z system:controller:statefulset-controller 2021-02-10T12:44:44Z system:controller:ttl-controller 2021-02-10T12:44:44Z system:kube-controller-manager 2021-02-10T12:44:44Z 以 deployment controller 为例： [root@k8s-master01 work]# kubectl describe clusterrole system:controller:deployment-controller Name: system:controller:deployment-controller Labels: kubernetes.io/bootstrapping=rbac-defaults Annotations: rbac.authorization.kubernetes.io/autoupdate: true PolicyRule: Resources Non-Resource URLs Resource Names Verbs --------- ----------------- -------------- ----- replicasets.apps [] [] [create delete get list patch update watch] replicasets.extensions [] [] [create delete get list patch update watch] events [] [] [create patch update] events.events.k8s.io [] [] [create patch update] pods [] [] [get list update watch] deployments.apps [] [] [get list update watch] deployments.extensions [] [] [get list update watch] deployments.apps/finalizers [] [] [update] deployments.apps/status [] [] [update] deployments.extensions/finalizers [] [] [update] deployments.extensions/status [] [] [update]
6.查看kube-controller-manager集群中当前的leader
[root@k8s-master01 work]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml apiVersion: v1 kind: Endpoints metadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master02_740a3931-ca6c-4872-aa5e-bd50b071074f","leaseDurationSeconds":15,"acquireTime":"2021-02-14T13:26:15Z","renewTime":"2021-02-14T14:03:20Z","leaderTransitions":1}' creationTimestamp: "2021-02-14T12:52:15Z" managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:control-plane.alpha.kubernetes.io/leader: {} manager: kube-controller-manager operation: Update time: "2021-02-14T12:52:15Z" name: kube-controller-manager namespace: kube-system resourceVersion: "110720" selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid: 0006cf35-91f8-4f16-a696-c5e969b0e233 可见，当前的leader为k8s-master02节点。 测试 kube-controller-manager 集群的高可用 停掉一个或两个节点的 kube-controller-manager 服务，观察其它节点的日志，看是否获取了 leader 权限。 比如停掉k8s-master02节点的kube-controller-manager 服务 [root@k8s-master02 ~]# systemctl stop kube-controller-manager [root@k8s-master02 ~]# ps -ef|grep kube-controller-manager root 732796 732615 0 22:06 pts/0 00:00:00 grep --color=auto kube-controller-manager 接着观察kube-controller-manager集群当前的leader情况 [root@k8s-master01 work]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml apiVersion: v1 kind: Endpoints metadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master01_cd88a6bf-279a-466d-95ce-52b489876095","leaseDurationSeconds":15,"acquireTime":"2021-02-14T14:06:10Z","renewTime":"2021-02-14T14:06:36Z","leaderTransitions":2}' creationTimestamp: "2021-02-14T12:52:15Z" managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:control-plane.alpha.kubernetes.io/leader: {} manager: kube-controller-manager operation: Update time: "2021-02-14T12:52:15Z" name: kube-controller-manager namespace: kube-system resourceVersion: "110959" selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid: 0006cf35-91f8-4f16-a696-c5e969b0e233 发现当前leader已经转移到k8s-master01节点上了！！
11.部署高可用 kube-scheduler 集群
该集群包含 3 个节点，启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用后，剩余节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。为保证通信安全，本文档先生成 x509 证书和私钥，kube-scheduler 在如下两种情况下使用该证书：与kube-apiserver 的安全端口通信;在安全端口(https，10251) 输出 prometheus 格式的 metrics；下面部署命令均在k8s-master01节点上执行，然后远程分发文件和执行命令。
1.创建kube-scheduler证书和私钥
1.创建证书签名请求： [root@k8s-master01 ~]# cd /opt/k8s/work [root@k8s-master01 work]# cat > kube-scheduler-csr.json <<EOF { "CN": "system:kube-scheduler", "hosts": [ "127.0.0.1", "172.31.46.28", "172.31.46.63", "172.31.46.67" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Hefei", "L": "Hefei", "O": "system:kube-scheduler", "OU": "4Paradigm" } ] } EOF 解释说明： hosts 列表包含所有 kube-scheduler 节点 IP； CN 和 O 均为 system:kube-scheduler，kubernetes 内置的 ClusterRoleBindings system:kube-scheduler 将赋予 kube-scheduler 工作所需的权限； 2.生成证书和私钥： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler [root@k8s-master01 work]# ls kube-scheduler*pem kube-scheduler-key.pem kube-scheduler.pem 3.将生成的证书和私钥分发到所有 master 节点： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" scp kube-scheduler*.pem root@${node_master_ip}:/etc/kubernetes/cert/ done
2.创建和分发kubeconfig文件
kube-scheduler 使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-scheduler 证书： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# kubectl config set-cluster kubernetes --certificate-authority=/opt/k8s/work/ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=kube-scheduler.kubeconfig [root@k8s-master01 work]# kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig [root@k8s-master01 work]# kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig [root@k8s-master01 work]# kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig 分发 kubeconfig 到所有 master 节点： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" scp kube-scheduler.kubeconfig root@${node_master_ip}:/etc/kubernetes/ done
3.创建kube-scheduler配置文件
[root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# cat >kube-scheduler.yaml.template <<EOF apiVersion: kubescheduler.config.k8s.io/v1alpha1 kind: KubeSchedulerConfiguration bindTimeoutSeconds: 600 clientConnection: burst: 200 kubeconfig: "/etc/kubernetes/kube-scheduler.kubeconfig" qps: 100 enableContentionProfiling: false enableProfiling: true hardPodAffinitySymmetricWeight: 1 healthzBindAddress: 0.0.0.0:10251 leaderElection: leaderElect: true metricsBindAddress: 0.0.0.0:10251 EOF 注意：这里的ip地址最好用0.0.0.0，不然执行"kubectl get cs"查看schedule的集群状态会是"Unhealthy" --kubeconfig：指定 kubeconfig 文件路径，kube-scheduler 使用它连接和验证 kube-apiserver； --leader-elect=true：集群运行模式，启用选举功能；被选为 leader 的节点负责处理工作，其它节点为阻塞状态； 替换模板文件中的变量： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for (( i=0; i < 3; i++ )) do sed -e "s/##NODE_MASTER_NAME##/${NODE_MASTER_NAMES[i]}/" -e "s/##NODE_MASTER_IP##/${NODE_MASTER_IPS[i]}/" kube-scheduler.yaml.template > kube-scheduler-${NODE_MASTER_IPS[i]}.yaml done 注意：NODE_NAMES 和 NODE_IPS 为相同长度的 bash 数组，分别为节点名称和对应的 IP； [root@k8s-master01 work]# ll kube-scheduler*.yaml -rw-r--r-- 1 root root 399 Feb 15 09:45 kube-scheduler-172.31.46.28.yaml -rw-r--r-- 1 root root 399 Feb 15 09:45 kube-scheduler-172.31.46.63.yaml -rw-r--r-- 1 root root 399 Feb 15 09:45 kube-scheduler-172.31.46.67.yaml 分发 kube-scheduler 配置文件到所有 master 节点： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" scp kube-scheduler-${node_master_ip}.yaml root@${node_master_ip}:/etc/kubernetes/kube-scheduler.yaml done 注意：重命名为 kube-scheduler.yaml;
4.创建 kube-scheduler systemd unit 模板文件
[root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# cat > kube-scheduler.service.template <<EOF [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] WorkingDirectory=${K8S_DIR}/kube-scheduler ExecStart=/opt/k8s/bin/kube-scheduler \ --config=/etc/kubernetes/kube-scheduler.yaml \ --bind-address=0.0.0.0 \
--secure-port=10259 \
--port=0 \ --tls-cert-file=/etc/kubernetes/cert/kube-scheduler.pem \ --tls-private-key-file=/etc/kubernetes/cert/kube-scheduler-key.pem \ --authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \ --client-ca-file=/etc/kubernetes/cert/ca.pem \ --requestheader-allowed-names="" \ --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \ --requestheader-extra-headers-prefix="X-Remote-Extra-" \ --requestheader-group-headers=X-Remote-Group \ --requestheader-username-headers=X-Remote-User \ --authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \ --logtostderr=true \ --v=2 Restart=always RestartSec=5 StartLimitInterval=0 [Install] WantedBy=multi-user.target EOF 1.为各节点创建和分发 kube-scheduler systemd unit 文件 替换模板文件中的变量，为各节点创建 systemd unit 文件： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for (( i=0; i < 3; i++ )) do sed -e "s/##NODE_MASTER_NAME##/${NODE_MASTER_NAMES[i]}/" -e "s/##NODE_MASTER_IP##/${NODE_MASTER_IPS[i]}/" kube-scheduler.service.template > kube-scheduler-${NODE_MASTER_IPS[i]}.service done 其中：NODE_NAMES 和 NODE_IPS 为相同长度的 bash 数组，分别为节点名称和对应的 IP； [root@k8s-master01 work]# ll kube-scheduler*.service
-rw-r--r-- 1 root root 983 Feb 15 09:49 kube-scheduler-172.31.46.28.service
-rw-r--r-- 1 root root 983 Feb 15 09:49 kube-scheduler-172.31.46.63.service
-rw-r--r-- 1 root root 983 Feb 15 09:49 kube-scheduler-172.31.46.67.service
2.分发 systemd unit 文件到所有 master 节点： [root@k8s-master01 work]# cd /opt/k8s/work [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" scp kube-scheduler-${node_master_ip}.service root@${node_master_ip}:/etc/systemd/system/kube-scheduler.service done
5.启动 kube-scheduler 服务
[root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" ssh root@${node_master_ip} "mkdir -p ${K8S_DIR}/kube-scheduler" ssh root@${node_master_ip} "systemctl daemon-reload && systemctl enable kube-scheduler && systemctl restart kube-scheduler" done 注意：启动服务前必须先创建工作目录； 检查服务运行状态 [root@k8s-master01 work]# source /opt/k8s/bin/environment.sh [root@k8s-master01 work]# for node_master_ip in ${NODE_MASTER_IPS[@]} do echo ">>> ${node_master_ip}" ssh root@${node_master_ip} "systemctl status kube-scheduler|grep Active" done