• python安装二进制k8s高可用 版本1.13.0


    一、所有安装包、脚本、脚本说明、下载链接:https://pan.baidu.com/s/1kHaesJJuMQ5cG-O_nvljtg 
    提取码:kkv6 
    
    
    二、脚本安装说明
    
    1、脚本说明:
    
    本实验为三台master组成高可用,另外一台单独安装node组件,一共四个节点;
    
    脚本里面的几个apiserver地址是写死的,所以部署时候用的ip不一样的话,需要替换;
    
    flanneld、coredns 文件里面ip是写死的,所以部署时候用的ip不一样的话,需要替换
    
    /root/ssl/目录下有几个shell脚本里面的一些信息也是写死的,需要替换。
    
    实验采用root进制安装,如果使用其他用户,所有命令前加上sudo;
    
    用到的技术点: shel、python、jinja2、ansible、playbook
    
    如果笔记本配置低的话,最好分开安装,先安装etcd集群,然后重启所有节点,再安装k8s集群。否则cpu会很高。导致到后面安装flananl总是无法启动
    
    测试过5遍,都可以搭建成功的,脚本不要动了,一动就容易出错,
    
    脚本耗时 2小时,如果笔记本配置高,安装完etcd,cpu不高的话,就不需要重启笔记本,那样可能需要1.5个小时
    
    
    2、项目架构:
    
    test1: 192.168.0.91    etcd、kubectl、kube-apiserver、kube-controller-manager、kube-scheduler、haproxy、keepalived ansible python
    
    test2: 192.168.0.92    etcd、kubectl、kube-apiserver、kube-controller-manager、kube-scheduler、haproxy、keepalived python
    
    test3: 192.168.0.93    etcd、kubectl、kube-apiserver、kube-controller-manager、kube-scheduler、haproxy、keepalived python
    
    test4: 192.168.0.94    docker、kubectl工具、kubelet组件、kube-proxy、flannel、coredns python jinja2
    
    
    3、环境配置
    
    参照本目录中环境配置篇进行操作
    
    说明:环境配置中包含:配置免密登录、安装ansible、关闭防火墙、selinux、swap、修改主机名、配置hosts解析、安装python、pip、重启
    
    
    
    4、test1节点创建目录
    
    mkdir -p /k8s/profile/
    
    mkdir -p /server/software/k8s/
    
    mkdir -p /root/ssl/
    
    mkdir -p /script/
    
    
    
    5、需要的文件放到/k8s/profile/目录下
    
    cd /k8s/profile/
    rz
    unzip profile.zip
    chmod +x /k8s/profile/*
    
    ls /k8s/profile/
    
    apiserver-to-kubelet.yaml     encryption-config.py                kube-apiserver.service.template.py.bak
    a.py                          encryption-config-template.py       kube-controller-manager.service.py
    audit-policy.yaml             etcd.service.py                     kube-controller-manager.service.template.py
    bootstrap-copy.sh             etcd.service.template.py            kube-flannel.yml.j2
    bootstrap-copy.sh.bak         flanneld.service.py                 kubelet.config.json.py
    bootstrap-kubeconfig.sh       flanneld.service.template.py        kubelet.config.json.template.py
    bootstrap-kubeconfig.sh.bak2  flannel_to_etcd.sh                  kubelet.service.py
    coredns.yaml.j2               haproxy.cfg                         kubelet.service.template.py
    csr-crb.yaml                  hosts                               kube-proxy.config.yaml.py
    daemon.json                   k8s.conf                            kube-proxy.config.yaml.template.py
    deploy_coredns.retry          keepalived-back.py                  kube-proxy.service
    deploy_coredns.yaml           keepalived-backup-template.py       kubernetes.conf
    deploy_flanneld.retry         keepalived-master.py                kube-scheduler.service
    deploy_flanneld.yaml          keepalived-master-template.py       profile.zip
    deploy_flanneld.yaml.bak      kube-apiserver.service.py           start-docker.sh
    docker.service                kube-apiserver.service.template.py  token-template.py
    
    
    
    
    6、安装包放到/server/software/k8s/下
    
    cd /server/software/k8s/
    
    rz
    
    ls /server/software/k8s/
    
    cfssl-certinfo_linux-amd64    docker-18.03.1-ce.tgz                         kubernetes                            README.md
    cfssljson_linux-amd64         docker-ce-18.03.0.ce-1.el7.centos.x86_64.rpm  kubernetes-server-linux-amd64.tar.gz  root@192.168.0.94
    cfssl_linux-amd64             etcd-v3.3.9-linux-amd64                       mk-docker-opts.sh
    cni-plugins-amd64-v0.7.1.tgz  etcd-v3.3.9-linux-amd64.tar.gz                pip-18.0.tar.gz
    coredns-1.2.2.tar.gz          flannel.tar                                   Python-3.6.5.tgz
    
    
    
    7、创建证书所需文件
    
    cd /root/ssl/
    rz ssl.zip
    unzip ssl.zip
    
    ls /root/ssl/
    
    admin.conf      ca.csr         flanneld.csr                 kube-controller-manager-csr.json  kubernetes-csr.json
    admin.csr       ca-csr.json    flanneld-csr.json            kube-proxy.csr                    kube-scheduler.csr
    admin-csr.json  etcd.csr       k8s.py                       kube-proxy-csr.json               kube-scheduler-csr.json
    ca-config.json  etcd-csr.json  kube-controller-manager.csr  kubernetes.csr                    ssl.zip
    
    
    
    
    8、脚本放到/script/目录下
    
    cd /script/
    
    rz k8s.py
    
    ls /script/
    
    k8s.py  pip-python.sh
    
    
    
    9、给辅助shell脚本授权
    
    chmod +x /k8s/profile/*
    
    
    
    9、打开主脚本函数
    
    去掉主脚本里面最后面几行函数注释
    
    
    
    10、开始执行
    
    python /script/k8s.py
    
    
    三、k8s.py脚本内容
    
    
    [root@test1 script]# cat /script/k8s.py 
    #!/usr/bin/python
    # -*- coding: utf-8 -*-
    
    from __future__ import print_function
    import os, sys, stat
    import shutil
    import tarfile
    import subprocess
    
    # 定义环境变量
    
    # 定义主机名
    NODE_NAME = subprocess.check_output(["hostname"], shell=True)
    NODE_NAME = str(NODE_NAME.decode('utf8').strip()).strip('b')
    
    # 定义主机ip
    NODE_IP = subprocess.check_output(["hostname -i | awk '{print $NF}'"], shell=True)
    NODE_IP = str(NODE_IP.decode('utf8').strip()).strip('b')
    
    #定义key
    ENCRYPTION_KEY = subprocess.check_output(["head -c 32 /dev/urandom | base64"], shell=True)
    ENCRYPTION_KEY = str(ENCRYPTION_KEY.decode('utf8').strip()).strip('b')
    
    SERVICE_CIDR = "10.254.0.0/16"
    CLUSTER_CIDR = "172.30.0.0/16"
    NODE_PORT_RANGE = "8000-30000"
    NODE_IPS = ['192.168.0.91', '192.168.0.92', '192.168.0.93']
    NODE_NAMES = ['test1', 'test2', 'test3','test4']
    MASTER_VIP = "192.168.0.235"
    KUBE_APISERVER = "https://192.168.0.235:8443"
    VIP_IF = "ens33"
    ETCD_ENDPOINTS = "https://192.168.0.91:2379,https://192.168.0.92:2379,https://192.168.0.93:2379"
    ETCD_NODES = "test1=https://192.168.0.91:2380,test2=https://192.168.0.92:2380,test3=https://192.168.0.93:2380"
    FLANNEL_ETCD_PREFIX = "/kubernetes/network/"
    CLUSTER_KUBERNETES_SVC_IP = "10.254.0.1"
    CLUSTER_DNS_SVC_IP = "10.254.0.2"
    CLUSTER_DNS_DOMAIN = "cluster.local."
    IFACE="ens33"
    
    
    
    def create_dir_kernel_parameters():
        print("创建目录、配置内核参数、安装依赖包")
        #所有节点创建目录
        subprocess.call(["time ansible k8s -m file -a 'path=/k8s/profile/ state=directory mode=0777'"], shell=True)
        subprocess.call(["time ansible k8s -m file -a 'path=/server/software/k8s/ state=directory mode=0777'"], shell=True)
        subprocess.call(["time ansible k8s -m file -a 'path=/root/ssl/ state=directory mode=0777'"], shell=True)
        subprocess.call(["time ansible k8s -m file -a 'path=/script/ state=directory mode=0777'"], shell=True)
        #配置内核参数
        subprocess.call(["ansible k8s -m shell -a 'iptables -P FORWARD ACCEPT'"], shell=True)
        subprocess.call(["time ansible k8s -m copy -a 'src=/k8s/profile/k8s.conf dest=/etc/sysctl.d/k8s.conf force=yes'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'sysctl --system'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'modprobe ip_vs'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'modprobe ip_vs_rr'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'modprobe ip_vs_wrr'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'modprobe ip_vs_sh'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'modprobe nf_conntrack_ipv4'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'lsmod | grep ip_vs'"], shell=True)
        #安装依赖包,  其实这一步只是配置node节点的kube-proxy步骤需要用到
        subprocess.call(["ansible k8s -m shell -a 'yum install -y conntrack ipvsadm ipset jq sysstat curl iptables libseccomp'"], shell=True)
    
    def install_cfssl():
        print("安装 cfssl 工具集")
        subprocess.call(["ansible k8s -m file -a 'path=/server/software/k8s/ state=directory mode=0777'"], shell=True)
        os.chdir('/server/software/k8s/')
        subprocess.call(["ansible test1 -m copy -a 'src=/server/software/k8s/cfssl-certinfo_linux-amd64  dest=/usr/local/bin/cfssl-certinfo force=yes'"], shell=True)
        subprocess.call(["ansible test1 -m copy -a 'src=/server/software/k8s/cfssl_linux-amd64  dest=/usr/local/bin/cfssl'"], shell=True)
        subprocess.call(["ansible test1 -m copy -a 'src=/server/software/k8s/cfssljson_linux-amd64  dest=/usr/local/bin/cfssljson'"], shell=True)
        os.chdir('/usr/local/bin/')
        os.chmod("cfssl-certinfo", stat.S_IXOTH)
        os.chmod("cfssl", stat.S_IXOTH)
        os.chmod("cfssljson", stat.S_IXOTH)
        print("successful")
    
    
    def create_root_ca():
        print("创建CA根证书")
        os.chdir('/root/ssl/')
        subprocess.call(["time ansible k8s -m file -a 'path=/root/ssl/ state=directory mode=0777'"], shell=True)
        subprocess.call(["cfssl gencert -initca ca-csr.json | cfssljson -bare ca"], shell=True)
        #创建用户
        subprocess.call(["time ansible k8s -m shell -a 'useradd k8s'"],shell=True)
        #创建目录
        subprocess.call(["time ansible k8s -m shell -a 'mkdir -p /etc/kubernetes/cert/ && chown -R k8s /etc/kubernetes'"],shell=True)
        subprocess.call(["time ansible k8s -m copy -a 'src=/root/ssl/ dest=/etc/kubernetes/cert/ force=yes'"], shell=True)
        print("successful")
    
    
    def distribute_kubectl():
        print("分发 所有 二进制文件、安装kubectl ")
        os.chdir('/server/software/k8s/')
        shutil.unpack_archive('kubernetes-server-linux-amd64.tar.gz')
        subprocess.call(["time ansible k8s -m file -a 'path=/opt/k8s/bin/ state=directory mode=0777'"], shell=True)
        subprocess.call(["time ansible k8s -m copy -a 'src=/server/software/k8s/kubernetes/server/bin/ dest=/opt/k8s/bin/ force=yes'"],shell=True)
        subprocess.call(["chmod +x /opt/k8s/bin/*"], shell=True)
        subprocess.call(["time ansible k8s -m copy -a 'src=/opt/k8s/bin/kubectl dest=/usr/local/bin/ force=yes'"], shell=True)
        subprocess.call(["time ansible k8s -m shell -a 'chmod +x /usr/local/bin/kubectl'"], shell=True)
        print("successful")
    
    def create_admin_ca():
        print("创建 admin 证书和私钥 ")
        os.chdir('/root/ssl/')
        subprocess.call(["ansible k8s -m shell -a 'chmod +x /etc/kubernetes/cert/*'"], shell=True)
        subprocess.call(["cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin"],shell=True)
        subprocess.call(["ansible k8s -m copy -a 'src=/root/ssl/admin.pem dest=/etc/kubernetes/cert/ force=yes'"], shell=True)
        subprocess.call(["ansible k8s -m copy -a 'src=/root/ssl/admin-key.pem dest=/etc/kubernetes/cert/ force=yes'"], shell=True)
        #给证书授权
        subprocess.call(["ansible k8s -m shell -a 'chmod +x /etc/kubernetes/cert/*'"], shell=True)
        print("successful")
    
    def create_admin_kubeconfig():
        print("创建admin kubeconfig 文件")
        os.chdir('/root/ssl/')
        subprocess.call(["kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem --embed-certs=true --server=https://192.168.0.235:8443 --kubeconfig=admin.conf"],shell=True)
        subprocess.call(["kubectl config set-credentials admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=admin.conf"],shell=True)
        subprocess.call(["kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=admin.conf"],shell=True)
        subprocess.call(["kubectl config use-context kubernetes --kubeconfig=admin.conf"],shell=True)
        subprocess.call(["ansible k8s -m file -a 'path=/root/.kube/ state=directory mode=0777'"], shell=True)
        subprocess.call(["ansible k8s -m copy -a 'src=/root/ssl/admin.conf dest=/root/.kube/config force=yes'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'chmod +x /root/.kube/*'"], shell=True)
        print("创建admin kubeconfig 文件完成")
    
    def install_etcd_cluster():
        print("部署 etcd 集群")
        # 解压和分发 etcd 二进制文件
        os.chdir('/server/software/k8s/')
        shutil.unpack_archive('etcd-v3.3.9-linux-amd64.tar.gz')
        subprocess.call(["time ansible k8s -m copy -a 'src=/server/software/k8s/etcd-v3.3.9-linux-amd64/ dest=/opt/k8s/bin/ force=yes'"],shell=True)
        subprocess.call(["time ansible k8s -m shell -a 'chmod +x /opt/k8s/bin/*'"],shell=True)
        subprocess.call(["time ansible k8s -m shell -a 'ln -s /opt/k8s/bin/etcdctl /usr/bin/etcdctl'"],shell=True)
        # 创建 etcd 证书和私钥
        os.chdir('/root/ssl/')
        subprocess.call(["cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd"],shell=True)
        subprocess.call(["time ansible k8s -m file -a 'path=/etc/etcd/cert/ state=directory mode=0777'"], shell=True)
        subprocess.call(["time ansible k8s -m copy -a 'src=/root/ssl/ dest=/etc/etcd/cert/ force=yes'"], shell=True)
        # 创建 etcd.service文件
        subprocess.call(["time ansible k8s -m file -a 'path=/k8s/profile/ state=directory mode=0777'"], shell=True)
        subprocess.call(["time ansible k8s -m copy -a 'src=/k8s/profile/ dest=/k8s/profile/ force=yes'"], shell=True)
        subprocess.call(["time ansible k8s -m shell -a 'python /k8s/profile/etcd.service.py'"],shell=True)
        #创建 etcd 数据目录
        subprocess.call(["time ansible k8s -m file -a 'path=/var/lib/etcd/ state=directory mode=0777'"], shell=True)
        subprocess.call(["time ansible k8s -m shell -a 'chmod +x /var/lib/etcd'"],shell=True)
        #启动 etcd 服务
        subprocess.call(["time ansible k8s -m shell -a 'systemctl daemon-reload'"],shell=True)
        subprocess.call(["time ansible k8s -m service -a 'name=etcd state=started'"],shell=True)
        subprocess.call(["ansible k8s -m service -a 'name=etcd enabled=yes'"],shell=True)
        #检查启动结果
        subprocess.call(["time ansible k8s -m shell -a 'systemctl status etcd'"],shell=True)
        #验证服务状态
        subprocess.call(["/opt/k8s/bin/etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file /etc/etcd/cert/etcd.pem --key-file /etc/etcd/cert/etcd-key.pem cluster-health"],shell=True)
        #说明:如果启动失败就把所有节点关机重启就即可解决
    
    def install_haproxy():
        print("安装haproxy")
        #安装软件
        subprocess.call(["ansible k8s -m shell -a 'yum install -y haproxy'"],shell=True)
        #配置和下发 haproxy 配置文件
        subprocess.call(["ansible k8s -m copy -a 'src=/k8s/profile/haproxy.cfg dest=/etc/haproxy force=yes'"], shell=True)
        #启动haproxy
        subprocess.call(["ansible k8s -m shell -a 'systemctl daemon-reload'"],shell=True)
        subprocess.call(["ansible k8s -m service -a 'name=haproxy state=started'"],shell=True)
        subprocess.call(["ansible k8s -m service -a 'name=haproxy enabled=yes'"],shell=True)
        #检查启动结果
        subprocess.call(["time ansible k8s -m shell -a 'systemctl status haproxy'"],shell=True)
    
    def install_keepalived():
        print("安装keepalived")
        #安装软件
        subprocess.call(["ansible k8s -m shell -a 'yum install -y keepalived'"],shell=True)
        #配置和下发 keepalived 配置文件
        subprocess.call(["ansible k8s -m copy -a 'src=/k8s/profile/ dest=/k8s/profile/ force=yes'"], shell=True)
        subprocess.call(["ansible test1 -m shell -a 'python /k8s/profile/keepalived-master.py'"],shell=True)
        subprocess.call(["ansible test0 -m shell -a 'python /k8s/profile/keepalived-back.py'"],shell=True)
        #起动keepalived 服务
        subprocess.call(["ansible k8s -m shell -a 'systemctl daemon-reload'"],shell=True)
        subprocess.call(["ansible k8s -m service -a 'name=keepalived state=started'"],shell=True)
        subprocess.call(["ansible k8s -m service -a 'name=keepalived enabled=yes'"],shell=True)
        #检查启动结果
        subprocess.call(["time ansible k8s -m shell -a 'systemctl status keepalived'"],shell=True)
        #测试是否能ping通vip
        subprocess.call(["time ansible k8s -m shell -a 'ping -c 1 192.168.0.235'"],shell=True)
        #查看 haproxy 状态页面 http://192.168.0.235:10080/status;账号密码:admin/123456
    
    def kube_apiserver():
        print("部署 kube-apiserver 组件")
        #创建 kubernetes 证书和私钥
        os.chdir('/root/ssl/')
        subprocess.call(["cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes"], shell=True)
        #生产service account key
        subprocess.call(["ansible test1 -m shell -a 'openssl genrsa -out /root/ssl/sa.key 2048'"], shell=True)
        subprocess.call(["ansible test1 -m shell -a 'openssl rsa -in /root/ssl/sa.key -pubout -out /root/ssl/sa.pub'"], shell=True)
        #将生成的证书、私钥、ervice account key 分发到所有节点
        subprocess.call(["ansible k8s -m copy -a 'src=/root/ssl/ dest=/etc/kubernetes/cert/ force=yes'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'chmod +x /etc/kubernetes/cert/'"], shell=True)
        #创建、分发加密配置文件
        subprocess.call(["ansible k8s -m shell -a 'python /k8s/profile/encryption-config.py'"],shell=True)
        subprocess.call(["ansible k8s -m copy -a 'src=/etc/kubernetes/encryption-config.yaml dest=/etc/kubernetes force=yes'"], shell=True)
        #创建、分发kube-apiserver文件
        subprocess.call(["ansible k8s -m copy -a 'src=/k8s/profile/kube-apiserver.service.py dest=/k8s/profile/ force=yes'"], shell=True)
        subprocess.call(["ansible k8s -m copy -a 'src=/k8s/profile/kube-apiserver.service.template.py dest=/k8s/profile/ force=yes'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'python /k8s/profile/kube-apiserver.service.py'"],shell=True)
        #起动kube-apiserver 服务
        subprocess.call(["ansible k8s -m shell -a 'systemctl daemon-reload'"],shell=True)
        subprocess.call(["ansible k8s -m service -a 'name=kube-apiserver state=started'"],shell=True)
        subprocess.call(["ansible k8s -m service -a 'name=kube-apiserver enabled=yes'"],shell=True)
        #检查 kube-apiserver服务 运行状态
        subprocess.call(["time ansible k8s -m shell -a 'systemctl status kube-apiserver'"],shell=True)
        #检查集群运行状态
        subprocess.call(["ansible k8s -m shell -a 'kubectl cluster-info'"],shell=True)
        subprocess.call(["ansible k8s -m shell -a 'kubectl get componentstatuses'"],shell=True)
        #授予 kubernetes 证书访问 kubelet API 的权限, 这句话感觉没什么用,到最后还是出现Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy)
        #所以在安装kubelet时候采取了绝招。
        subprocess.call(["ansible k8s -m shell -a 'kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernete'"],shell=True)
        print("上面报错忽略,不要乱改脚本了,之前这里只是改成了test1,集群就无法启动了")
        subprocess.call(["ansible k8s -m shell -a 'sleep 60s'"], shell=True)
    
    def kube_controller_manager():
        print("部署高可用 kube-controller-manager 集群")
        #创建 kubernetes 证书和私钥
        os.chdir('/root/ssl/')
        subprocess.call(["cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager"], shell=True)
        #将生成的证书和私钥分发到所有节点
        subprocess.call(["ansible k8s -m copy -a 'src=/root/ssl/ dest=/etc/kubernetes/cert/ force=yes'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'chmod +x /etc/kubernetes/cert/'"], shell=True)
        #创建和分发 kubeconfig 文件
        subprocess.call(["kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem --embed-certs=true --server=https://192.168.0.235:8443 --kubeconfig=kube-controller-manager.kubeconfig"], shell=True)
        subprocess.call(["kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig"], shell=True)
        subprocess.call(["kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig"], shell=True)
        subprocess.call(["kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig"], shell=True)
        subprocess.call(["ansible k8s -m copy -a 'src=/root/ssl/kube-controller-manager.kubeconfig dest=/etc/kubernetes/ force=yes'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'chmod +x /etc/kubernetes/*'"], shell=True)
        #创建、分发kube-controller-manager.service
        subprocess.call(["ansible k8s -m copy -a 'src=/k8s/profile/kube-controller-manager.service.template.py  dest=/k8s/profile/ force=yes'"], shell=True)
        subprocess.call(["ansible k8s -m copy -a 'src=/k8s/profile/kube-controller-manager.service.py dest=/k8s/profile/ force=yes'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'python /k8s/profile/kube-controller-manager.service.py'"],shell=True)
        #创建日志目录
        subprocess.call(["ansible k8s -m file -a 'path=/var/log/kubernetes/ state=directory mode=0777'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'chmod +x /var/log/kubernetes/'"], shell=True)
        #启动 kube-controller-manager 服务
        subprocess.call(["ansible k8s -m shell -a 'systemctl daemon-reload'"],shell=True)
        subprocess.call(["ansible k8s -m service -a 'name=kube-controller-manager state=started'"],shell=True)
        subprocess.call(["ansible k8s -m service -a 'name=kube-controller-manager enabled=yes'"],shell=True)
        #检查 kube-controller-manager 运行状态
        subprocess.call(["time ansible k8s -m shell -a 'systemctl status kube-controller-manager'"],shell=True)
        #查看leader节点
        subprocess.call(["kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml"],shell=True)
    
    def install_kube_scheduler():
        print("部署高可用 kube-scheduler 集群")
        #创建 kube-scheduler 证书和私钥
        os.chdir('/root/ssl/')
        subprocess.call(["cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler"], shell=True)
        #将生成的证书和私钥分发到所有节点
        subprocess.call(["ansible k8s -m copy -a 'src=/root/ssl/kube-scheduler.pem dest=/etc/kubernetes/cert/ force=yes'"], shell=True)
        subprocess.call(["ansible k8s -m copy -a 'src=/root/ssl/kube-scheduler-key.pem dest=/etc/kubernetes/cert/ force=yes'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'chmod +x /etc/kubernetes/cert/'"], shell=True)
        #创建和分发 kubeconfig 文件
        subprocess.call(["kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem --embed-certs=true --server=https://192.168.0.235:8443 --kubeconfig=kube-scheduler.kubeconfig"], shell=True)
        subprocess.call(["kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig"], shell=True)
        subprocess.call(["kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig"], shell=True)
        subprocess.call(["kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig"], shell=True)
        subprocess.call(["ansible k8s -m copy -a 'src=/root/ssl/kube-scheduler.kubeconfig dest=/etc/kubernetes/ force=yes'"], shell=True)
        subprocess.call(["ansible k8s -m shell -a 'chmod +x /etc/kubernetes/*'"], shell=True)
        #创建和分发kube-scheduler.service
        subprocess.call(["ansible k8s -m copy -a 'src=/k8s/profile/kube-scheduler.service dest=/etc/systemd/system/ force=yes'"], shell=True)
        #启动kube-scheduler 服务
        subprocess.call(["ansible k8s -m shell -a 'systemctl daemon-reload'"],shell=True)
        subprocess.call(["ansible k8s -m service -a 'name=kube-scheduler state=started'"],shell=True)
        subprocess.call(["ansible k8s -m service -a 'name=kube-scheduler enabled=yes'"],shell=True)
        #检查 kube-scheduler 运行状态
        subprocess.call(["time ansible k8s -m shell -a 'systemctl status kube-scheduler'"],shell=True)
        #查看leader节点
        subprocess.call(["kubectl get endpoints kube-scheduler --namespace=kube-system -o yaml"],shell=True)
    
    def environmental_configuration():
        print("下面开始单独安装worker节点,worker节点主机名:test4")
        print("配置环境")
        #所有节点创建目录
        subprocess.call(["time ansible test4 -m file -a 'path=/k8s/profile/ state=directory mode=0777'"], shell=True)
        subprocess.call(["time ansible test4 -m file -a 'path=/server/software/k8s/ state=directory mode=0777'"], shell=True)
        subprocess.call(["time ansible test4 -m file -a 'path=/root/ssl/ state=directory mode=0777'"], shell=True)
        subprocess.call(["time ansible test4 -m file -a 'path=/script/ state=directory mode=0777'"], shell=True)
        #加载ipvs内核参数,其实这一步在安装kube-proxy那一步时执行即可
        subprocess.call(["ansible test4 -m shell -a 'iptables -P FORWARD ACCEPT'"], shell=True)
        subprocess.call(["time ansible test4 -m copy -a 'src=/k8s/profile/k8s.conf dest=/etc/sysctl.d/k8s.conf force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'sysctl --system'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'modprobe ip_vs'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'modprobe ip_vs_rr'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'modprobe ip_vs_wrr'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'modprobe ip_vs_sh'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'modprobe nf_conntrack_ipv4'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'lsmod | grep ip_vs'"], shell=True)
        #安装依赖包,  其实这一步只是配置node节点的kube-proxy步骤需要用到
        subprocess.call(["ansible test4 -m shell -a 'yum install -y conntrack ipvsadm ipset jq sysstat curl iptables libseccomp'"], shell=True)
        #创建k8s群组
        subprocess.call(["ansible test4 -m group -a 'name=k8s state=present'"], shell=True)
        #创建k8s用户
        subprocess.call(["ansible test4 -m user -a 'name=k8s group=k8s'"], shell=True)
    
    def install_docker():
        print("部署docker")
        #移除自带安装包
        subprocess.call(["ansible test4 -m shell -a 'yum remove -y docker-ce docker-ce-selinux container-selinux'"],shell=True)
        #下发docker安装包
        subprocess.call(["ansible test4 -m copy -a 'src=/server/software/k8s/docker-ce-18.03.0.ce-1.el7.centos.x86_64.rpm dest=/server/software/k8s/ force=yes'"], shell=True)
        #安装docker
        subprocess.call(["ansible test4 -m shell -a 'chdir=/server/software/k8s/ yum install -y docker-ce-*.rpm'"],shell=True)
        #配置docker加速
        subprocess.call(["ansible test4 -m copy -a 'src=/k8s/profile/daemon.json dest=/etc/docker/ force=yes'"], shell=True)
        #启动docker
        subprocess.call(["ansible test4 -m shell -a 'systemctl start docker'"],shell=True)
        subprocess.call(["ansible test4 -m shell -a 'systemctl enable docker'"],shell=True)
    
    def install_kubelet():
        print("部署 kubelet 组件")
        os.chdir('/root/ssl/')
        #创建工作目录和日志目录
        subprocess.call(["ansible test4 -m file -a 'path=/var/lib/kubelet/ state=directory mode=0777'"], shell=True)
        subprocess.call(["ansible test4 -m file -a 'path=/var/log/kubernetes/ state=directory mode=0777'"], shell=True)
        #配置admin-config
        subprocess.call(["ansible test4 -m file -a 'path=/root/.kube/ state=directory mode=0777'"], shell=True)
        subprocess.call(["ansible test4 -m copy -a 'src=/root/ssl/admin.conf dest=/root/.kube/config force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'chmod +x /root/.kube/*'"], shell=True)
        #配置kubectl工具
        subprocess.call(["ansible test4 -m copy -a 'src=/opt/k8s/bin/kubectl dest=/usr/local/bin/ force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'chmod +x /usr/local/bin/kubectl'"], shell=True)
        #配置kubeadm工具, 注意:在test1上操作
        subprocess.call(["ansible test1 -m copy -a 'src=/opt/k8s/bin/kubeadm dest=/usr/local/bin/ force=yes'"], shell=True)
        subprocess.call(["ansible test1 -m shell -a 'chmod +x /usr/local/bin/kubeadm'"], shell=True)
        #创建 kubelet bootstrap kubeconfig 文件,包含创建token
        subprocess.call(["ansible test1 -m shell -a 'chmod +x /k8s/profile/bootstrap-kubeconfig.sh'"], shell=True)
        subprocess.call(["ansible test1 -m shell -a 'sh /k8s/profile/bootstrap-kubeconfig.sh'"], shell=True)
        #分发kubelet bootstrap kubeconfig 文件到所有节点
        subprocess.call(["ansible test1 -m shell -a 'chmod +x /k8s/profile/bootstrap-copy.sh'"], shell=True)
        subprocess.call(["ansible test1 -m shell -a 'sh /k8s/profile/bootstrap-copy.sh'"], shell=True)
        print("上面报错忽略,没有影响")
        subprocess.call(["ansible test4 -m shell -a 'sleep 30s'"],shell=True)
        #配置 bootstrap RBAC 权限
        subprocess.call(["kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers"],shell=True)
        #查看 kubeadm 为各节点创建的 token;之前好好的,后来查看报错
        #subprocess.call(["ansible test1 -m shell -a 'kubeadm token list --kubeconfig ~/.kube/config'"], shell=True)
        #分发ca证书到worker 节点
        subprocess.call(["ansible test4 -m copy -a 'src=/root/ssl/ca.pem dest=/etc/kubernetes/cert/ force=yes'"], shell=True)
        #创建 kubelet 参数配置文件
        subprocess.call(["ansible test4 -m copy -a 'src=/k8s/profile/kubelet.config.json.template.py dest=/k8s/profile/ force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m copy -a 'src=/k8s/profile/kubelet.config.json.py dest=/k8s/profile/ force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'python /k8s/profile/kubelet.config.json.py'"],shell=True)
        #分发kubelet二进制文件
        subprocess.call(["time ansible test4 -m copy -a 'src=/opt/k8s/bin/kubelet dest=/opt/k8s/bin/ force=yes'"], shell=True)
        subprocess.call(["time ansible test4 -m shell -a 'chmod +x /opt/k8s/bin/kubelet'"], shell=True)
        #创建和分发kubelet启动文件
        subprocess.call(["ansible test4 -m copy -a 'src=/k8s/profile/kubelet.service.template.py dest=/k8s/profile/ force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m copy -a 'src=/k8s/profile/kubelet.service.py dest=/k8s/profile/ force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'python /k8s/profile/kubelet.service.py'"],shell=True)
        #给kunelet授权操作pod权限,包括进入pod等操作,真正解决 Forbidden (user=kubernetes,)问题
        subprocess.call(["ansible test4 -m copy -a 'src=/k8s/profile/apiserver-to-kubelet.yaml dest=/k8s/profile/apiserver-to-kubelet.yaml force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'kubectl create -f /k8s/profile/apiserver-to-kubelet.yaml'"],shell=True)
        #启动 kubelet 服务
        subprocess.call(["ansible test4 -m shell -a 'systemctl daemon-reload'"],shell=True)
        subprocess.call(["ansible test4 -m service -a 'name=kubelet state=started'"],shell=True)
        subprocess.call(["ansible test4 -m service -a 'name=kubelet enabled=yes'"],shell=True)
        #检查启动结果
        subprocess.call(["ansible test4 -m shell -a 'systemctl status kubelet'"],shell=True)
        #设置集群角色
        test4=subprocess.check_output(["kubectl get nodes | grep test4 | awk '{print $1}'"], shell=True)
        test4=test4.decode('utf8').strip()
        subprocess.call(['kubectl','label','nodes',test4,'node-role.kubernetes.io/node='])
        print("查看csr,如果是Approved,Issued状态,下面这个报错忽略: No resources found error: resource name may not be empty")
        subprocess.call(["ansible test4 -m shell -a 'kubectl get csr'"],shell=True)
        subprocess.call(["ansible test4 -m shell -a 'sleep 60s'"],shell=True)
    
    def approve_csr():
        print("自动 approve CSR 请求")
        subprocess.call(["time ansible test4 -m copy -a 'src=/k8s/profile/csr-crb.yaml dest=/k8s/profile/csr-crb.yaml force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'kubectl apply -f /k8s/profile/csr-crb.yaml'"],shell=True)
        print("No resources found error: resource name may not be empty 这个报错忽略,不影响的,")
        subprocess.call(["ansible test4 -m shell -a 'sleep 30s'"],shell=True)
        #等待一段时间(1-10 分钟),所有节点的 CSR 都被自动 approve;看了下时间,这次刚好等了10分钟
        subprocess.call(["ansible test4 -m shell -a 'sleep 300s'"],shell=True)
        #查看csr
        subprocess.call(["ansible test4 -m shell -a 'kubectl get csr'"],shell=True)
        #所有节点均 ready
        subprocess.call(["ansible test4 -m shell -a 'kubectl get nodes'"],shell=True)
        #查看kube-controller-manager 为各 node 生成了 kubeconfig 文件和公私钥
        subprocess.call(["ansible test4 -m shell -a 'ls -l /etc/kubernetes/kubelet.kubeconfig'"],shell=True)
        subprocess.call(["ansible test4 -m shell -a 'ls -l /etc/kubernetes/cert/|grep kubelet'"],shell=True)
        #查看kubelet 提供的 API 接口
        subprocess.call(["ansible test4 -m shell -a 'yum install net-tools -y'"],shell=True)
        subprocess.call(["ansible test4 -m shell -a 'netstat -lnpt|grep kubelet'"],shell=True)
    
    def install_kube_proxy():
        print("部署 kube-proxy 组件")
        #创建工作目录和日志目录
        subprocess.call(["ansible test4 -m file -a 'path=/var/lib/kube-proxy/ state=directory mode=0777'"], shell=True)
        subprocess.call(["ansible test4 -m file -a 'path=/var/log/kubernetes/ state=directory mode=0777'"], shell=True)
        subprocess.call(["ansible test4 -m file -a 'path=/var/log/kubernetes/ owner=k8s group=k8s mode=0777'"], shell=True)
        #安装依赖包
        subprocess.call(["ansible test4 -m shell -a 'yum install -y conntrack ipvsadm ipset jq sysstat curl iptables libseccomp'"], shell=True)
        #加载 ip_vs 内核模块
        subprocess.call(["ansible test4 -m shell -a 'iptables -P FORWARD ACCEPT'"], shell=True)
        subprocess.call(["time ansible test4 -m copy -a 'src=/k8s/profile/k8s.conf dest=/etc/sysctl.d/k8s.conf force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'sysctl --system'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'modprobe ip_vs'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'modprobe ip_vs_rr'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'modprobe ip_vs_wrr'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'modprobe ip_vs_sh'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'modprobe nf_conntrack_ipv4'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'lsmod | grep ip_vs'"], shell=True)
        #创建 kube-proxy 证书和私钥
        os.chdir('/root/ssl/')
        subprocess.call(["cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy"], shell=True)
        #将生成的证书和私钥分发到test4节点
        subprocess.call(["ansible test4 -m copy -a 'src=/root/ssl/kube-proxy.pem dest=/etc/kubernetes/cert/ force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m copy -a 'src=/root/ssl/kube-proxy-key.pem dest=/etc/kubernetes/cert/ force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'chmod +x /etc/kubernetes/cert/'"], shell=True)
        #创建和分发 kubeconfig 文件
        subprocess.call(["kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem --embed-certs=true --server=https://192.168.0.235:8443 --kubeconfig=kube-proxy.kubeconfig"], shell=True)
        subprocess.call(["kubectl config set-credentials system:kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig"], shell=True)
        subprocess.call(["kubectl config set-context system:kube-proxy --cluster=kubernetes --user=system:kube-proxy --kubeconfig=kube-proxy.kubeconfig"], shell=True)
        subprocess.call(["kubectl config use-context system:kube-proxy --kubeconfig=kube-proxy.kubeconfig"], shell=True)
        subprocess.call(["ansible test4 -m copy -a 'src=/root/ssl/kube-proxy.kubeconfig dest=/etc/kubernetes/ force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'chmod +x /etc/kubernetes/*'"], shell=True)
        #创建 kube-proxy 配置文件
        subprocess.call(["ansible test4 -m copy -a 'src=/k8s/profile/kube-proxy.config.yaml.template.py dest=/k8s/profile/ force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m copy -a 'src=/k8s/profile/kube-proxy.config.yaml.py dest=/k8s/profile/ force=yes'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'python /k8s/profile/kube-proxy.config.yaml.py'"],shell=True)
        #分发kube-proxy二进制文件
        subprocess.call(["time ansible test4 -m copy -a 'src=/opt/k8s/bin/kube-proxy dest=/opt/k8s/bin/ force=yes'"], shell=True)
        subprocess.call(["time ansible test4 -m shell -a 'chmod +x /opt/k8s/bin/kube-proxy'"], shell=True)
        #创建和分发kube-proxy.service
        subprocess.call(["ansible test4 -m copy -a 'src=/k8s/profile/kube-proxy.service dest=/etc/systemd/system/kube-proxy.service force=yes'"], shell=True)
        #启动kube-proxy 服务
        subprocess.call(["ansible test4 -m shell -a 'systemctl daemon-reload'"],shell=True)
        subprocess.call(["ansible test4 -m service -a 'name=kube-proxy state=started'"],shell=True)
        subprocess.call(["ansible test4 -m service -a 'name=kube-proxy enabled=yes'"],shell=True)
        #检查 kube-proxy 运行状态
        subprocess.call(["time ansible test4 -m shell -a 'systemctl status kube-proxy'"],shell=True)
    
    def install_flanneld():
        print("部署 flannel 网络")
        #关闭防火墙
        subprocess.call(["ansible test4 -m service -a 'name=firewalld state=stopped'"],shell=True)
        subprocess.call(["time ansible test4 -m shell -a 'systemctl disable firewalld'"],shell=True)
        #安装jinja2模块
        subprocess.call(["ansible test4 -m shell -a 'pip3 install jinja2'"], shell=True)
        #通过playbook安装flannel,在playbook中定义了通过kubectl安装flanneld;playbook中定义了安装在test4节点上
        subprocess.call(["ansible-playbook /k8s/profile/deploy_flanneld.yaml"], shell=True)
        #等(1-10)分钟查看flannel pod
        subprocess.call(["ansible test4 -m shell -a 'sleep 100s'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'kubectl get pod -n kube-system'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'kubectl get nodes'"], shell=True)
    
    def install_coredns():
        print("安装coredns")
        print("下面报错忽略,没有影响。这个错误: No package jq available.Error: Nothing to donon-zero return code")
        subprocess.call(["ansible test4 -m shell -a 'yum install -y jq 2> /dev/null'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'sleep 30s'"], shell=True)
        #通过playbook安装coredns,在playbook中定义了通过kubectl安装coredns;playbook中定义了安装在test4节点上
        subprocess.call(["ansible-playbook /k8s/profile/deploy_coredns.yaml"], shell=True)
        #等(1-10)分钟查看coredns pod
        subprocess.call(["ansible test4 -m shell -a 'sleep 60s'"], shell=True)
        subprocess.call(["ansible test4 -m shell -a 'kubectl get pod -n kube-system'"], shell=True)
        #测试coredns请参照https://www.cnblogs.com/effortsing/p/10312081.html 最后部分
    
    def func_list():
        #install_cfssl()
        #create_root_ca()
        #distribute_kubectl()
        #create_admin_ca()
        #create_admin_kubeconfig()
        #install_etcd_cluster()
        #install_haproxy()
        #install_keepalived()
        #kube_apiserver()
        #kube_controller_manager()
        #install_kube_scheduler()
        #下面开始单独安装worker节点,worker节点地址:192.168.0.94
        #environmental_configuration()
        #install_docker()
        #install_kubelet()
        #approve_csr()
        #install_kube_proxy()
        #install_flanneld()
        #install_coredns()
    def main():
        func_list()
    if __name__ == '__main__':
        main()
  • 相关阅读:
    Maven笔记之面试题合集
    Maven笔记之核心概念及常用命令
    UML中的关系
    RocketMq核心概念
    linux安装rocketMq(包括安装maven,JDK)
    linux安装JDK,配置环境变量
    ASP.NET Core读取appsettings.json配置文件信息
    ASP.NET Core获取客户端IP地址
    ASP.NET Core根据环境切换NLog配置
    ASP.NET Core使用NLog记录日志
  • 原文地址:https://www.cnblogs.com/effortsing/p/10362135.html
Copyright © 2020-2023  润新知