Centos7二进制部署k8s-v1.20.2 ipvs版本(controller-manager、kube-scheduler、kubelet)

一、部署kube-controller-manager

获取最新更新以及文章用到的软件包，请移步点击：查看更新

1、创建csr请求文件

cat > kube-controller-manager-csr.json << EOF
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.112.131",
      "192.168.112.132",
      "192.168.112.133",
      "192.168.112.134",
      "192.168.112.135",
      "192.168.112.136",
      "192.168.112.130"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "Sichuan",
        "L": "Chengdu",
        "O": "system:kube-controller-manager",
        "OU": "system"
      }
    ]
}
EOF

生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

注：
hosts 列表包含所有 kube-controller-manager 节点 IP；
CN 为 system:kube-controller-manager、O 为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限

2、创建kube-controller-manager的kubeconfig

设置集群参数
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.112.130:7443 --kubeconfig=kube-controller-manager.kubeconfig
设置客户端认证参数
kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig
设置上下文参数
kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
设置默认上下文
kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

3、创建配置文件

cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--secure-port=10257 \
  --bind-address=127.0.0.1 \
  --kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \
  --service-cluster-ip-range=10.255.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --allocate-node-cidrs=true \
  --cluster-cidr=10.0.0.0/16 \
  --experimental-cluster-signing-duration=175200h \
  --root-ca-file=/opt/kubernetes/ssl/ca.pem \
  --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --leader-elect=true \
  --feature-gates=RotateKubeletServerCertificate=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --horizontal-pod-autoscaler-use-rest-clients=true \
  --horizontal-pod-autoscaler-sync-period=10s \
  --tls-cert-file=/opt/kubernetes/ssl/kube-controller-manager.pem \
  --tls-private-key-file=/opt/kubernetes/ssl/kube-controller-manager-key.pem \
  --use-service-account-credentials=true \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/logs \
  --v=2"
EOF

4、创建启动文件

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

5、同步相关文件到各个节点

cp kube-controller-manager*.pem /opt/kubernetes/ssl/
cp kube-controller-manager.kubeconfig /opt/kubernetes/cfg
scp kube-controller-manager*.pem root@192.168.112.132:/opt/kubernetes/ssl/
scp kube-controller-manager.kubeconfig  root@192.168.112.132:/opt/kubernetes/cfg

6、启动服务

systemctl daemon-reload 
systemctl enable kube-controller-manager
systemctl start kube-controller-manager
systemctl status kube-controller-manager

二、部署kube-scheduler

1、创建csr请求文件

cat > kube-scheduler-csr.json << EOF
{
    "CN": "system:kube-scheduler",
    "hosts": [
      "127.0.0.1",
      "192.168.112.131",
      "192.168.112.132",
      "192.168.112.133",
      "192.168.112.134",
      "192.168.112.135",
      "192.168.112.136",
      "192.168.112.130"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "Sichuan",
        "L": "Chengdu",
        "O": "system:kube-scheduler",
        "OU": "system"
      }
    ]
}
EOF

生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

注：
hosts 列表包含所有 kube-scheduler 节点 IP；
CN 为 system:kube-scheduler、O 为 system:kube-scheduler，kubernetes 内置的 ClusterRoleBindings system:kube-scheduler 将赋予 kube-scheduler 工作所需的权限。

2、创建kube-scheduler的kubeconfig

设置集群参数
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.112.130:7443 --kubeconfig=kube-scheduler.kubeconfig
设置客户端认证参数
kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig
设置上下文参数
kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
设置默认上下文
kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

3、创建配置文件

cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--address=127.0.0.1 
--kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig 
--leader-elect=true 
--alsologtostderr=true 
--logtostderr=false 
--log-dir=/opt/kubernetes/logs 
--v=2"
EOF

4、创建服务启动文件

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

5、同步相关文件到各个节点

cp kube-scheduler*.pem /opt/kubernetes/ssl/
cp kube-scheduler.kubeconfig /opt/kubernetes/cfg
scp kube-scheduler*.pem root@192.168.112.132:/opt/kubernetes/ssl/
scp kube-scheduler.kubeconfig  root@192.168.112.132:/opt/kubernetes/cfg
scp /usr/lib/systemd/system/kube-scheduler.service root@192.168.112.132:/usr/lib/systemd/system/

6、启动服务

systemctl daemon-reload
systemctl enable kube-scheduler
systemctl start kube-scheduler
systemctl status kube-scheduler

三、部署kubelet

1、生成kubelet-bootstrap文件

#创建kubelet-bootstrap.kubeconfig
BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /opt/kubernetes/cfg/token.csv)

#设置集群参数
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.112.131:6443 --kubeconfig=kubelet-bootstrap.kubeconfig

#设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap.kubeconfig

#设置上下文参数
kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig

#设置默认上下文
kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig

#创建角色绑定
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

2、创建配置文件

cat > kubelet.json << EOF
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/opt/kubernetes/ssl/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "address": "192.168.112.131",                      #注：kubelete.json配置文件address改为各个节点的ip地址
  "port": 10250,
  "readOnlyPort": 10255,
  "cgroupDriver": "systemd",                         #如果docker的驱动为cgroupfs，处修改为cgroupfs。此处设置很重要，否则后面node节点无法加入到集群，写入配置文件时，记得去掉文中的中文注释，容易引起报错
  "hairpinMode": "promiscuous-bridge",
  "serializeImagePulls": false,
  "featureGates": {
    "RotateKubeletClientCertificate": true,
    "RotateKubeletServerCertificate": true
  },
  "clusterDomain": "cluster.local.",
  "clusterDNS": ["10.255.0.2"]
}
EOF

3、创建启动文件

cat > /usr/lib/systemd/system/kubelet.service << EOF 
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/opt/kubernetes/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \
  --bootstrap-kubeconfig=/opt/kubernetes/cfg/kubelet-bootstrap.kubeconfig \
  --cert-dir=/opt/kubernetes/ssl \
  --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
  --config=/opt/kubernetes/cfg/kubelet.json \
  --network-plugin=cni \
  --pod-infra-container-image=k8s.gcr.io/pause:3.2 \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/logs \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

注：
–hostname-override：显示名称，集群中唯一
–network-plugin：启用CNI
–kubeconfig：空路径，会自动生成，后面用于连接apiserver
–bootstrap-kubeconfig：首次启动向apiserver申请证书
–config：配置参数文件
–cert-dir：kubelet证书生成目录
–pod-infra-container-image：管理Pod网络容器的镜像

4、同步相关文件到各个节点

cd /root/TLS/k8s/kubernetes/server/bin
cp kubelet /opt/kubernetes/bin/
cd /root/TLS/k8s
cp kubelet-bootstrap.kubeconfig kubelet.json /opt/kubernetes/cfg/

scp kubelet-bootstrap.kubeconfig kubelet.json /opt/kubernetes/cfg/
cd /root/TLS/k8s/kubernetes/server/bin
scp kubelet root@192.168.112.132:/opt/kubernetes/bin/
scp /usr/lib/systemd/system/kubelet.service root@192.168.112.132:/usr/lib/systemd/system/

scp /opt/kubernetes/cfg/token.csv root@192.168.112.133:/opt/kubernetes/cfg/
scp /opt/kubernetes/ssl/ca* root@192.168.112.133:/opt/kubernetes/ssl/

5、启动服务

mkdir /opt/kubernetes/kubelet
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet

6、批准kubelet证书申请并加入集群

# 查看kubelet证书请求
kubectl get csr
NAME                                                   AGE    SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--K6M4G7bjhk8A   6m3s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

# 批准申请
kubectl certificate approve node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--K6M4G7bjhk8A

# 查看节点
kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
clihouse01   NotReady <none>   16h   v1.20.2
clihouse02   NotReady <none>   16h   v1.20.2
clihouse03   NotReady <none>   16h   v1.20.2
clihouse04   NotReady <none>   16h   v1.20.2