k8s1.15.2 二进制部署

1. 实验架构

主机名	IP地址
hdss7-11.host.com	10.4.7.11
hdss7-12.host.com	10.4.7.12
hdss7-21.host.com	10.4.7.21
hdss7-22.host.com	10.4.7.22
hdss7-23.host.com	10.4.7.23
hdss7-24.host.com	10.4.7.24
hdss7-200.host.com	10.4.7.200

2 实验准备工作

2.1 准备虚拟机

7台VM, 每台2c2g

配置vmware网段为 10.4.7.0, 子网掩码为 255.255.255.0

配置vmware的网关为 10.4.7.254

将vmnet8 网卡的地址配置为 10.4.7.1 ，子网掩码为 255.255.255.0

设置接口的自动跃点数为10，默认自动跃点

在 /etc/sysconfig/network-scripts/ifcfg-eth0 配置网卡

TYPE=Ethernet
BOOTPROTO=static
NAME=eth0
DEVICE=eth0
ONBOOT=yes
IPADDR=10.4.7.11
NETMASK=255.255.255.0
GATEWAY=10.4.7.254
DNS1=10.4.7.254
DNS2=223.5.5.5

重启网卡

systemctl restart network

2.2 配置基础环境

配置阿里的源，并安装epel-release

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum makecache
yum -y install epel-release

关闭SElinux 和 firewalld

systemctl stop firewalld
systemctl disable firewalld
sed -i 's/SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config

安装必要的工具

yum -y install wget net-tools telnet tree nmap sysstat lrzsz dos2unix bind-utils

2.3 DNS服务初始化

10.4.7.11 上

2.3.1 安装bind

yum -y install bind

2.3.2 配置主配置文件

修改主配置文件 /etc/named.conf

options {
    listen-on port 53 { 10.4.7.11; }; //监听地址
    directory   "/var/named";
    dump-file   "/var/named/data/cache_dump.db";
    statistics-file "/var/named/data/named_stats.txt";
    memstatistics-file "/var/named/data/named_mem_stats.txt";
    recursing-file  "/var/named/data/named.recursing";
    secroots-file   "/var/named/data/named.secroots";
    allow-query     { any; };  //哪一些客户端可以通过当前主机这台DNS查到DNS关系
  forwarders      { 10.4.7.254; };  //上一层DNS, 如果当前DNS无法查询, 则往上一层找， 即网关
    recursion yes;   // DNS服务采用递归算法给客户端提供DNS查询
    dnssec-enable yes;
    dnssec-validation yes;
    /* Path to ISC DLV key */
    bindkeys-file "/etc/named.root.key";
    managed-keys-directory "/var/named/dynamic";
    pid-file "/run/named/named.pid";
    session-keyfile "/run/named/session.key";
};

2.3.3 配置区域配置文件

新建区域配置文件 /etc/named/named.rfc1912.zones

zone "host.com" IN {
        type master;
        file "host.com.zone";
        allow-update { 10.4.7.11; };
};
zone "od.com" IN {
        type master;
        file "nb.com.zone";
        allow-update { 10.4.7.11; };
};

2.3.4 配置区域数据文件

2.3.4.1 新建区域数据文件 /var/named/host.com.zone

ORIGIN host.com.
$TTL 600    ; 10 minutes
@   IN SOA  dns.host.com.  dnsadmin.host.com.  (
                 2020062201   ; serial
                 10800      ; refresh (3 hours)
                 900        ; retry (15 minutes)
                 604800     ; expire (1 week)
                 86400      ; minimun (1 day)
                 )
            NS  dns.host.com.
$TTL 60 ; 1 minute
dns        A      10.4.7.11
hdss7-11           A      10.4.7.11 
hdss7-12           A      10.4.7.12
hdss7-21           A      10.4.7.21
hdss7-22           A      10.4.7.22
hdss7-23           A      10.4.7.23
hdss7-24           A      10.4.7.24
hdss7-200          A      10.4.7.20

2.3.4.2 新建区域数据文件 /var/named/od.com.zone

$ORIGIN od.com.
$TTL 600    ; 10 minutes
@   IN SOA  dns.od.com.  dnsadmin.od.com.  (
                 2020062203   ; serial
                 10800      ; refresh (3 hours)
                 900        ; retry (15 minutes)
                 604800     ; expire (1 week)
                 86400      ; minimun (1 day)
                 )
            NS  dns.od.com.
$TTL 60 ; 1 minute
dns        A      10.4.7.11
harbor             A      10.4.7.200

2.3.4.3 启动named服务

systemctl start named
systemctl enable named

2.3.4.4 验证dns服务是否可用

dig -t A hdss7-11.host.com @10.4.7.11 +short

2.3.5 设置本地dns服务（10.4.7.11）为客户端提供服务

10.4.7.11 / 10.4.7.12 / 10.4.7.21 / 10.4.7.22 / 10.4.7.200 / 10.4.7.23 / 10.4.7.24 上操作

sed -i 's/DNS1=10.4.7.254/DNS1=10.4.7.11/' /etc/sysconfig/network-scripts/ifcfg-eth0
systemctl restart network

验证dns

dig -t A hdss7-11.host.com @10.4.7.11 +short
dig -t A hdss7-12.host.com @10.4.7.11 +short
dig -t A hdss7-21.host.com @10.4.7.11 +short
dig -t A hdss7-22.host.com @10.4.7.11 +short
dig -t A hdss7-200.host.com @10.4.7.11 +short

ping baidu.com

ping hdss7-11.host.com

2.3.6 配置短域名

sed  -i '2isearch host.com' /etc/resolv.conf

ping hdss7-11

2.3.7 配置windows 的Vmnet8网卡自己配置的DNS进行解析

设置vmnet8的DNS地址

将vmnet8的dns地址修改为 10.4.7.11

启用telent客户端

控制画板-程序-启用或关闭Windows功能

telnet 10.4.7.11 53

测试dns配置

ping hdss7-11.host.com

2.4 准备证书签发环境

在主机 hdss7-200.host.com（10.4.7.200）上

一般证书签发环境使用openssl或者CFSSL

2.4.1 安装CFSSL

在 https://pkg.cfssl.org/ 下载 cfssl_linux-amd64 / cfssljson_linux-amd64/cfssl-certinfo_linux-amd64 , 保存到 /root/plugins/

cd /root/plugins/
cp cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
cp cfssljson_linux-amd64 /usr/bin/cfssljson
cp cfssl_linux-amd64 /usr/bin/cfssl
chmod +x /usr/bin/cfssl*

2.4.2 创建根证书（CA证书）的JSON配置文件

/opt/certs/ca-scr.json

{
    "CN": "OldboyEdu",
    "hosts": [
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ],
    "ca": {
        "expiry": "175200h"
    }
}

说明：

CN: Common Name 浏览器使用该字段验证网站是否合法，一般写域名

C: Country 国家

ST: State 州，省

L: LOcality 地区，城市

O: Organization Name 组织名称，公司名称

OU：Organization Unit Name 组织单位名称，公司部门

2.4.3 生成CA 证书和私钥

cfssl gencert -initca ca-scr.json |cfssljson -bare ca

部署docker环境

hdss7-200.host.com（10.4.7.200）

hdss7-21.host.com（10.4.7.21）

hdss7-22.host.com（10.4.7.22）

hdss7-23.host.com（10.4.7.23）

hdss7-24.host.com（10.4.7.24）

2.5.1 安装

curl -fsSL https://get.docker.com |bash -s docker --mirror Aliyun

2.5.2 配置

hdss7-21.host.com（10.4.7.21）

mkdir /etc/docker
mkdir -p /data/docker
cat > /etc/docker/daemon.json <<EOF
{
"graph": "/data/docker",
"storage-driver": "overlay2",
"insecure-registries": ["registry.access.redhat.com", "quay.io", "harbor.od.com"],
"registry-mirrors": ["https://q2gr04ke.mirror.aliyuncs.com"],
"bip": "172.7.21.1/24",
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true
}
EOF

hdss7-22.host.com（10.4.7.22）

mkdir /etc/docker
mkdir -p /data/docker
cat > /etc/docker/daemon.json <<EOF
{
"graph": "/data/docker",
"storage-driver": "overlay2",
"insecure-registries": ["registry.access.redhat.com", "quay.io", "harbor.od.com"],
"registry-mirrors": ["https://q2gr04ke.mirror.aliyuncs.com"],
"bip": "172.7.21.1/24",
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true
}
EOF

hdss7-200.host.com（10.4.7.200）

mkdir /etc/docker
mkdir -p /data/docker
cat > /etc/docker/daemon.json <<EOF
{
"graph": "/data/docker",
"storage-driver": "overlay2",
"insecure-registries": ["registry.access.redhat.com", "quay.io", "harbor.od.com"],
"registry-mirrors": ["https://q2gr04ke.mirror.aliyuncs.com"],
"bip": "172.7.21.1/24",
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true
}
EOF

hdss7-23.host.com（10.4.7.23）

mkdir /etc/docker
mkdir -p /data/docker
cat > /etc/docker/daemon.json <<EOF
{
"graph": "/data/docker",
"storage-driver": "overlay2",
"insecure-registries": ["registry.access.redhat.com", "quay.io", "harbor.od.com"],
"registry-mirrors": ["https://q2gr04ke.mirror.aliyuncs.com"],
"bip": "172.7.21.1/24",
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true
}
EOF

hdss7-24.host.com（10.4.7.24）

mkdir /etc/docker
mkdir -p /data/docker
cat > /etc/docker/daemon.json <<EOF
{
"graph": "/data/docker",
"storage-driver": "overlay2",
"insecure-registries": ["registry.access.redhat.com", "quay.io", "harbor.od.com"],
"registry-mirrors": ["https://q2gr04ke.mirror.aliyuncs.com"],
"bip": "172.7.21.1/24",
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true
}
EOF

2.5.3 启动

systemctl start docker
systemctl enable docker

2.6 部署docker镜像仓库 harbor

hdss7-200.host.com（10.4.7.200）

2.6.1 安装docker-compose

https://docs.docker.com/compose/install/

while true;
do  
curl -L "https://github.com/docker/compose/releases/download/1.26.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose; 
if [[ $? -eq 0 ]];then 
break; 
fi; 
done

2.6.2 下载 harbor 1.10.3

从官网下载

https://github.com/goharbor/harbor/releases/download/v1.10.3/harbor-offline-installer-v1.10.3.tgz

2.6.3 使用nginx反向代理 harbor.od.com

使用域名 harbor.od.com 访问harbor

yum -y install nginx

配置虚拟主机 od.harbor.com.conf

server {
    listen 80;
    server_name harbor.od.com

    location / {
        proxy_pass http://127.0.0.1:180;
    }

}

启动nginx

systemctl start nginx
systemctl enable nginx

2.6.4 修改区域数据文件 /var/named/od.com.zone

10.4.7.11 上操作

$ORIGIN od.com.
$TTL 600    ; 10 minutes
@   IN SOA  dns.od.com.  dnsadmin.od.com.  (
                 2020062202   ; serial
                 10800      ; refresh (3 hours)
                 900        ; retry (15 minutes)
                 604800     ; expire (1 week)
                 86400      ; minimun (1 day)
                 )
            NS  dns.od.com.
$TTL 60 ; 1 minute
dns        A      10.4.7.11
harbor             A      10.4.7.200

重启named服务

systemctl restart named

dig -t A harbor.od.com +short

2.6.5 配置harbor

mkdir /opt/src
tar -xf harbor-offline-installer-v1.10.3.tgz -C /opt
mv /opt/harbor/ /opt/harbor-v1.10.3
ln -s /opt/harbor-v1.10.3/ /opt/harbor   # 便于以后版本的升级
cd /opt/harbor
./prepare

修改配置文件 harbor.yml

hostname: harbor.od.com

http:
  port: 180
  
data_volume: /data/harbor

log:
  level: info
  local:
    rotate_count: 50
    rotate_size: 200M
    location: /data/harbor/logs

创建目录 /data/harbor/logs

mkdir -p /data/harbor/logs

2.6.6 启动harbor

/opt/harbor/install.sh

docker ps

cd /opt/harbor
docker-compose ps

浏览器输入网址 http://harbor.od.com

2.6.7 新建项目public

2.6.8 上传镜像

docker pull nginx:1.7.9
docker tag nginx:1.7.9 harbor.od.com/public/nginx:v1.7.9
docker login harbor.od.com
docker push harbor.od.com/public/nginx:v1.7.9

这个时候会有报错 413 Request Entity Too Large，解决方案是配置nginx主配置文件中最大上传文件的大小限制 ( client_max_body_size 50000m;)

vim /etc/nginx/nginx.conf
http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;
    client_max_body_size 50000m;
systemctl restart nginx

docker push harbor.od.com/public/nginx:v1.7.9

3. 部署 k8s master 节点服务

3.1 部署etcd集群

集群规划

主机名	角色	ip
hdss7-12.host.com	etcd lead	10.4.7.12
hdss7-21.host.com	etcd follow	10.4.7.21
hdss7-22.host.com	etcd follow	10.4.7.22

mkdir /opt/src

3.1.1 创建基于根证书的config配置文件

在 10.4.7.200 上生成

vim /opt/certs/ca-config.json
{
    "signing": {
        "default": {
            "expiry": "175200h"
        },
        "profiles": {
            "server": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

3.1.2 创建etcd的请求配置文件

在 10.4.7.200 上生成

vim /opt/certs/etcd-peer-csr.json
{
    "CN": "k8s-etcd",
    "hosts": [
        "10.4.7.11",
        "10.4.7.12",
        "10.4.7.21",
        "10.4.7.22"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

3.1.3 生成etcd证书和私钥

在 10.4.7.200 上生成

cd /opt/certs
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json |cfssljson -bare etcd-peer

检查生成的证书，私钥

3.1.4 创建etcd用户

在 10.4.7.12 / 10.4.7.21 / 10.4.7.22 操作

useradd -s /sbin/nologin -M etcd

3.1.5 下载软件，解压，做软连接

在 10.4.7.12 / 10.4.7.21 / 10.4.7.22 操作

https://github.com/etcd-io/etcd/releases/download/v3.1.20/etcd-v3.1.20-linux-amd64.tar.gz

cd /opt
tar -xf etcd-v3.1.20-linux-amd64.tar.gz
mv etcd-v3.1.20-linux-amd64 etcd-v3.1.20
ln -s /opt/etcd-v3.1.20 /opt/etcd

3.1.6 创建目录，拷贝证书文件

在 10.4.7.12 / 10.4.7.21 / 10.4.7.22 操作

mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server

3.1.7 拷贝生成的证书文件

在 10.4.7.12 / 10.4.7.21 / 10.4.7.22 操作

cd /opt/etcd/certs
scp 10.4.7.200:/opt/certs/ca.pem .
scp 10.4.7.200:/opt/certs/etcd-peer.pem .
scp 10.4.7.200:/opt/certs/etcd-peer-key.pem .

3.1.9 创建etcd服务启动脚本

vi /opt/etcd/etcd-server-startup.sh

10.4.7.12

#!/bin/sh
./etcd --name etcd-server-7-12 
       --data-dir /data/etcd/etcd-server 
       --listen-peer-urls https://10.4.7.12:2380 
       --listen-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 
       --quota-backend-bytes 8000000000 
       --initial-advertise-peer-urls https://10.4.7.12:2380 
       --advertise-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 
       --initial-cluster  etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 
       --ca-file ./certs/ca.pem 
       --cert-file ./certs/etcd-peer.pem 
       --key-file ./certs/etcd-peer-key.pem 
       --client-cert-auth  
       --trusted-ca-file ./certs/ca.pem 
       --peer-ca-file ./certs/ca.pem 
       --peer-cert-file ./certs/etcd-peer.pem 
       --peer-key-file ./certs/etcd-peer-key.pem 
       --peer-client-cert-auth 
       --peer-trusted-ca-file ./certs/ca.pem 
       --log-output stdout

10.4.7.21

#!/bin/sh
./etcd --name etcd-server-7-21 
       --data-dir /data/etcd/etcd-server 
       --listen-peer-urls https://10.4.7.21:2380 
       --listen-client-urls https://10.4.7.21:2379,http://127.0.0.1:2379 
       --quota-backend-bytes 8000000000 
       --initial-advertise-peer-urls https://10.4.7.21:2380 
       --advertise-client-urls https://10.4.7.21:2379,http://127.0.0.1:2379 
       --initial-cluster  etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 
       --ca-file ./certs/ca.pem 
       --cert-file ./certs/etcd-peer.pem 
       --key-file ./certs/etcd-peer-key.pem 
       --client-cert-auth  
       --trusted-ca-file ./certs/ca.pem 
       --peer-ca-file ./certs/ca.pem 
       --peer-cert-file ./certs/etcd-peer.pem 
       --peer-key-file ./certs/etcd-peer-key.pem 
       --peer-client-cert-auth 
       --peer-trusted-ca-file ./certs/ca.pem 
       --log-output stdout

10.4.7.22

#!/bin/sh
./etcd --name etcd-server-7-22 
       --data-dir /data/etcd/etcd-server 
       --listen-peer-urls https://10.4.7.22:2380 
       --listen-client-urls https://10.4.7.22:2379,http://127.0.0.1:2379 
       --quota-backend-bytes 8000000000 
       --initial-advertise-peer-urls https://10.4.7.22:2380 
       --advertise-client-urls https://10.4.7.22:2379,http://127.0.0.1:2379 
       --initial-cluster  etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 
       --ca-file ./certs/ca.pem 
       --cert-file ./certs/etcd-peer.pem 
       --key-file ./certs/etcd-peer-key.pem 
       --client-cert-auth  
       --trusted-ca-file ./certs/ca.pem 
       --peer-ca-file ./certs/ca.pem 
       --peer-cert-file ./certs/etcd-peer.pem 
       --peer-key-file ./certs/etcd-peer-key.pem 
       --peer-client-cert-auth 
       --peer-trusted-ca-file ./certs/ca.pem 
       --log-output stdout

3.1.10 授权目录权限

在 10.4.7.12 / 10.4.7.21 / 10.4.7.22 操作

chown -R etcd.etcd /opt/etcd-v3.1.20
chown -R etcd.etcd /data/etcd/
chown -R etcd.etcd /data/logs/etcd-server/

3.1.11 安装supervisor软件

在 10.4.7.12 / 10.4.7.21 / 10.4.7.22 操作

yum -y install supervisor
systemctl start supervisord.service 
systemctl enable supervisord.service

3.1.12 .创建supervisor配置

vi /etc/supervisord.d/etcd-server.ini

10.4.7.12

[program:etcd-server-7-12]
command=/opt/etcd/etcd-server-startup.sh                        ; the program (relative uses PATH, can take args)
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/opt/etcd                                             ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=30                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd                                                       ; setuid to this UNIX account to run the program
redirect_stderr=true                                            ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log           ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)

10.4.7.21

[program:etcd-server-7-21]
command=/opt/etcd/etcd-server-startup.sh                        ; the program (relative uses PATH, can take args)
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/opt/etcd                                             ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=30                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd                                                       ; setuid to this UNIX account to run the program
redirect_stderr=true                                            ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log           ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)

10.4.7.22

[program:etcd-server-7-22]
command=/opt/etcd/etcd-server-startup.sh                        ; the program (relative uses PATH, can take args)
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/opt/etcd                                             ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=30                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd                                                       ; setuid to this UNIX account to run the program
redirect_stderr=true                                            ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log           ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)

3.1.13 启动etcd服务并检查

在 10.4.7.12 / 10.4.7.21 / 10.4.7.22 操作

supervisorctl update
supervisorctl status
netstat -lntup|grep etcd

3.1.14 检查集群状态

在 10.4.7.12 / 10.4.7.21 / 10.4.7.22 任意一台操作

/opt/etcd/etcdctl cluster-health

3.2 部署kube-apiserver集群

主机名	角色	ip地址
hdss7-21.host.com	kube-apiserver	10.4.7.21
hdss7-22.host.com	kube-apiserver	10.4.7.22

3.2.1下载软件，解压，做软连接

10.4.7.21 / 10.4.7.22 上操作

https://dl.k8s.io/v1.15.2/kubernetes-server-linux-amd64.tar.gz

cd /opt
tar -xf kubernetes-server-linux-amd64-v1.15.2.tar.gz
mv kubernetes kubernetes-1.15.2
ln -s /opt/kubernetes-1.15.2/ /opt/kubernetes
cd kubernetes
rm -rf kubernetes-src.tar.gz
cd server/bin
rm -f *.tar
rm -f *_tag

3.2.2 签发client证书

10.4.7.200 上操作

cd /opt/certs/
vi /opt/certs/client-csr.json
{
    "CN": "k8s-node",
    "hosts": [
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json |cfssljson -bare client

检查生成的证书文件

10.4.7.200 上操作

cd /opt/certs/
ll

3.2.3 签发kube-apiserver证书

10.4.7.200 上操作

cd /opt/certs/
vi /opt/certs/apiserver-csr.json
{
    "CN": "k8s-apiserver",
    "hosts": [
        "127.0.0.1",
        "192.168.0.1",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local",
        "10.4.7.10",
        "10.4.7.21",
        "10.4.7.22",
        "10.4.7.23"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json |cfssljson -bare apiserver

检查生成的证书文件

10.4.7.200 上操作

cd /opt/certs/
ll

3.2.4 拷贝证书和私钥

10.4.7.21 / 10.4.7.22 上操作

cd /opt/kubernetes/server/bin/
mkdir cert
cd cert
scp hdss7-200:/opt/certs/ca.pem .                                                                                                                                       100% 1363     1.5MB/s   00:00    
scp hdss7-200:/opt/certs/ca-key.pem .
scp hdss7-200:/opt/certs/client.pem .                                                                                                                                     100% 1614   711.4KB/s   00:00    
scp hdss7-200:/opt/certs/client-key.pem .                                                                                                                                 100% 1675     1.8MB/s   00:00    
scp hdss7-200:/opt/certs/apiserver.pem .                                                                                                                                100% 1614   964.6KB/s   00:00    
scp hdss7-200:/opt/certs/apiserver-key.pem .

3.2.5 创建配置

10.4.7.21 / 10.4.7.22 上操作

apiserver启动需要日志审计功能，否则起不来

cd /opt/kubernetes/server/bin/
mkdir conf
cd conf/
cat > audit.yaml <<EOF
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]

  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"

  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"
EOF

3.2.6 创建apiserver启动脚本

10.4.7.21 / 10.4.7.22 上操作

vi /opt/kubernetes/server/bin/kube-apiserver.sh
#!/bin/bash
./kube-apiserver 
  --apiserver-count 2 
  --audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log 
  --audit-policy-file ./conf/audit.yaml 
  --authorization-mode RBAC 
  --client-ca-file ./cert/ca.pem 
  --requestheader-client-ca-file ./cert/ca.pem 
  --enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota 
  --etcd-cafile ./cert/ca.pem 
  --etcd-certfile ./cert/client.pem 
  --etcd-keyfile ./cert/client-key.pem 
  --etcd-servers https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 
  --service-account-key-file ./cert/ca-key.pem 
  --service-cluster-ip-range 192.168.0.0/16 
  --service-node-port-range 3000-29999 
  --target-ram-mb=1024 
  --kubelet-client-certificate ./cert/client.pem 
  --kubelet-client-key ./cert/client-key.pem 
  --log-dir  /data/logs/kubernetes/kube-apiserver 
  --tls-cert-file ./cert/apiserver.pem 
  --tls-private-key-file ./cert/apiserver-key.pem 
  --v 2

3.2.7 授权和创建目录

chmod +x /opt/kubernetes/server/bin/kube-apiserver.sh
mkdir -p /data/logs/kubernetes/kube-apiserver

3.2.8 创建supervisor配置

vi /etc/supervisord.d/kube-apiserver.ini

10.4.7.21

[program:kube-apiserver-7-21]
command=/opt/kubernetes/server/bin/kube-apiserver.sh            ; the program (relative uses PATH, can take args)
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                            ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=30                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                       ; setuid to this UNIX account to run the program
redirect_stderr=true                                            ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log        ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)

10.4.7.22

[program:kube-apiserver-7-22]
command=/opt/kubernetes/server/bin/kube-apiserver.sh            ; the program (relative uses PATH, can take args)
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                            ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=30                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                       ; setuid to this UNIX account to run the program
redirect_stderr=true                                            ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log        ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)

3.2.9 启动服务并检查

supervisorctl update
supervisorctl status
netstat -nltup|grep kube-api

3.3.部署四层反向代理

主机名	角色	IP地址	VIP地址
hdss7-11.host.com	L4	10.4.7.11	10.4.7.10
hdss7-12.host.com	L4	10.4.7.12	10.4.7.10

3.3.1 配置nginx反向代理

10.4.7.11 / 10.4.7.12 上操作

yum -y install nginx

nginx配置

/etc/nginx/nginx.conf

stream {
    upstream kube-apiserver {
        server 10.4.7.21:6443     max_fails=3 fail_timeout=30s;
        server 10.4.7.22:6443     max_fails=3 fail_timeout=30s;
    }
    server {
        listen 7443;
        proxy_connect_timeout 2s;
        proxy_timeout 900s;
        proxy_pass kube-apiserver;
    }
}

启动服务

systemctl start nginx
systemctl enable nginx

3.3.2 配置高可用（keepalived）

10.4.7.11 / 10.4.7.12 上操作

yum -y install keepalived

3.3.2.1 check_port.sh

/etc/keepalived/check_port.sh

#!/bin/bash
#keepalived 监控端口脚本
#使用方法：
#在keepalived的配置文件中
#vrrp_script check_port {#创建一个vrrp_script脚本,检查配置
#    script "/etc/keepalived/check_port.sh 6379" #配置监听的端口
#    interval 2 #检查脚本的频率,单位（秒）
#}
CHK_PORT=$1
if [ -n "$CHK_PORT" ];then
        PORT_PROCESS=`ss -lnt|grep $CHK_PORT|wc -l`
        if [ $PORT_PROCESS -eq 0 ];then
                echo "Port $CHK_PORT Is Not Used,End."
                exit 1
        fi
else
        echo "Check Port Cant Be Empty!"
fi

添加可执行权限

chmod +x /etc/keepalived/check_port.sh

3.3.2.2 hdss7-11.host.com(10.4.7.11) keepalive主

/etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id 10.4.7.11

}

vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 7443"
    interval 2
    weight -20
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 251
    priority 100
    advert_int 1
    mcast_src_ip 10.4.7.11
    nopreempt

    authentication {
        auth_type PASS
        auth_pass 11111111
    }
    track_script {
         chk_nginx
    }
    virtual_ipaddress {
        10.4.7.10
    }
}

注意！！！！ nopreempt 这个配置表示非抢占式，如果主keepalived节点（10.4.7.11）端口挂掉，则vip会飘到备keepalived节点（10.4.7.12），当主keepalived节点起来， vip不会回到主keepalived

启动服务

systemctl start keepalived
systemctl enable keepalived

3.3.2.3 hdss7-12.host.com(10.4.7.12) keepalive备

/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id 10.4.7.12
}
vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 7443"
    interval 2
    weight -20
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 251
    mcast_src_ip 10.4.7.12
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 11111111
    }
    track_script {
        chk_nginx
    }
    virtual_ipaddress {
        10.4.7.10
    }
}

注意！！！！ nopreempt 这个配置表示非抢占式，如果主keepalived节点端口挂掉，则vip会飘到从keepalived节点，当主keepalived节点起来， vip不会回到主keepalived

启动服务

systemctl start keepalived
systemctl enable keepalived

3.3.2.4 检查vip

netstat -ntulp |grep :7443
ip a s eth0

3.3.2.5 制造一场事故

如果 hdss7-11.host.com(10.4.7.11) keepalive主节点挂了

nginx -s stop

vip就会启动到 hdss7-12.host.com(10.4.7.12) keepalive备份节点上了

ip a s eth0

此时把 hdss7-11.host.com(10.4.7.11) keepalive主节点 的7443端口起来

nginx
netstat -ntulp|grep :7443

观察到vip还是没有回到 hdss7-11.host.com(10.4.7.11) keepalive主节点

ip a s eth0

这是因为hdss7-11.host.com(10.4.7.11) keepalive主节点 配置了 nopreempt 这个配置

nopreempt 这个配置表示非抢占式，如果主keepalived节点端口挂掉，则vip会飘到从keepalived节点，当主keepalived节点起来， vip不会回到主keepalived

在生产中VIP是不允许随便切换的，如果要切换，必须保证配置已经正确了，等到流量较低的时候进行切换

切换的时候需要重启 hdss7-11.host.com(10.4.7.11) keepalive主节点的 keepalived服务

systemctl restart keepalived.service

3.4.部署controller-manager

主机名	角色	IP地址
hdss7-21.host.com	controller-manager	10.4.7.21
hdss7-22.host.com	controller-manager	10.4.7.22

3.4.1 创建启动脚本

vi /opt/kubernetes/server/bin/kube-controller-manager.sh
#!/bin/sh
./kube-controller-manager 
  --cluster-cidr 172.7.0.0/16 
  --leader-elect true 
  --log-dir /data/logs/kubernetes/kube-controller-manager 
  --master http://127.0.0.1:8080 
  --service-account-private-key-file ./cert/ca-key.pem 
  --service-cluster-ip-range 192.168.0.0/16 
  --root-ca-file ./cert/ca.pem 
  --v 2

3.4.2 授权文件权限，创建目录

chmod +x /opt/kubernetes/server/bin/kube-controller-manager.sh 
mkdir -p /data/logs/kubernetes/kube-controller-manager

3.4.3 创建supervisor配置

vi /etc/supervisord.d/kube-conntroller-manager.ini

10.4.7.21

[program:kube-controller-manager-7-21]
command=/opt/kubernetes/server/bin/kube-controller-manager.sh                     ; the program (relative uses PATH, can take args)
numprocs=1                                                                        ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                                              ; directory to cwd to before exec (def no cwd)
autostart=true                                                                    ; start at supervisord start (default: true)
autorestart=true                                                                  ; retstart at unexpected quit (default: true)
startsecs=30                                                                      ; number of secs prog must stay running (def. 1)
startretries=3                                                                    ; max # of serial start failures (default 3)
exitcodes=0,2                                                                     ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                                   ; signal used to kill process (default TERM)
stopwaitsecs=10                                                                   ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                                         ; setuid to this UNIX account to run the program
redirect_stderr=true                                                              ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controller.stdout.log  ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                                      ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                                          ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                                       ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                                       ; emit events on stdout writes (default false)

10.4.7.22

[program:kube-controller-manager-7-22]
command=/opt/kubernetes/server/bin/kube-controller-manager.sh                     ; the program (relative uses PATH, can take args)
numprocs=1                                                                        ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                                              ; directory to cwd to before exec (def no cwd)
autostart=true                                                                    ; start at supervisord start (default: true)
autorestart=true                                                                  ; retstart at unexpected quit (default: true)
startsecs=30                                                                      ; number of secs prog must stay running (def. 1)
startretries=3                                                                    ; max # of serial start failures (default 3)
exitcodes=0,2                                                                     ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                                   ; signal used to kill process (default TERM)
stopwaitsecs=10                                                                   ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                                         ; setuid to this UNIX account to run the program
redirect_stderr=true                                                              ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controller.stdout.log  ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                                      ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                                          ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                                       ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                                       ; emit events on stdout writes (default false)

3.4.4 启动服务并检查

supervisorctl update
supervisorctl status

3.5.部署kube-scheduler

主机名	角色	IP地址
hdss7-21.host.com	kube-scheduler	10.4.7.21
hdss7-22.host.com	kube-scheduler	10.4.7.22

3.5.1 创建启动脚本

vi /opt/kubernetes/server/bin/kube-scheduler.sh
#!/bin/sh
./kube-scheduler 
  --leader-elect  
  --log-dir /data/logs/kubernetes/kube-scheduler 
  --master http://127.0.0.1:8080 
  --v 2

3.5.2 授权文件权限，创建目录

chmod +x  /opt/kubernetes/server/bin/kube-scheduler.sh
mkdir -p /data/logs/kubernetes/kube-scheduler

3.5.3 创建supervisor配置

vi /etc/supervisord.d/kube-scheduler.ini

10.4.7.21

[program:kube-scheduler-7-21]
command=/opt/kubernetes/server/bin/kube-scheduler.sh                     ; the program (relative uses PATH, can take args)
numprocs=1                                                               ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                                     ; directory to cwd to before exec (def no cwd)
autostart=true                                                           ; start at supervisord start (default: true)
autorestart=true                                                         ; retstart at unexpected quit (default: true)
startsecs=30                                                             ; number of secs prog must stay running (def. 1)
startretries=3                                                           ; max # of serial start failures (default 3)
exitcodes=0,2                                                            ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                          ; signal used to kill process (default TERM)
stopwaitsecs=10                                                          ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                                ; setuid to this UNIX account to run the program
redirect_stderr=true                                                     ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                             ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                                 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                              ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                              ; emit events on stdout writes (default false)

10.4.7.22

[program:kube-scheduler-7-22]
command=/opt/kubernetes/server/bin/kube-scheduler.sh                     ; the program (relative uses PATH, can take args)
numprocs=1                                                               ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                                     ; directory to cwd to before exec (def no cwd)
autostart=true                                                           ; start at supervisord start (default: true)
autorestart=true                                                         ; retstart at unexpected quit (default: true)
startsecs=30                                                             ; number of secs prog must stay running (def. 1)
startretries=3                                                           ; max # of serial start failures (default 3)
exitcodes=0,2                                                            ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                          ; signal used to kill process (default TERM)
stopwaitsecs=10                                                          ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                                ; setuid to this UNIX account to run the program
redirect_stderr=true                                                     ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                             ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                                 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                              ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                              ; emit events on stdout writes (default false)

3.5.4 启动服务并检查

supervisorctl update
supervisorctl status

3.6 检查master节点

ln -s /opt/kubernetes/server/bin/kubectl /usr/bin/kubectl
kubectl get cs

4. 部署Node节点服务

4.1 部署kubelet

主机名	角色	IP地址
hdss7-21.host.com	kubelet	10.4.7.21
hdss7-22.host.com	kubelet	10.4.7.22
hdss7-23.host.com	kubelet	10.4.7.23
hdss7-24.host.com	kubelet	10.4.7.24

4.1.1 签发kubelet证书

10.4.7.200 上操作

vi /opt/certs/kubelet-csr.json
{
    "CN": "k8s-kubelet",
    "hosts": [
    "127.0.0.1",
    "10.4.7.10",
    "10.4.7.21",
    "10.4.7.22",
    "10.4.7.23",
    "10.4.7.24",
    "10.4.7.25",
    "10.4.7.26",
    "10.4.7.27",
    "10.4.7.28"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

生成证书

cd /opt/certs
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssljson -bare kubelet

检查证书

ll /opt/certs

4.1.2 拷贝证书到各运算节点

10.4.7.21 / 10.4.7.22 / 10.4.7.23 / 10.4.7.24 操作

分发证书到 10.4.7.21 / 10.4.7.22

cd /opt/kubernetes/server/bin/cert
scp hdss7-200:/opt/certs/ca.pem .
scp hdss7-200:/opt/certs/kubelet.pem .
scp hdss7-200:/opt/certs/kubelet-key.pem .

4.1.3 创建配置

cd /opt/kubernetes/conf

1. set-cluster

kubectl config set-cluster myk8s 
--certificate-authority=/opt/kubernetes/server/bin/cert/ca.pem 
--embed-certs=true 
--server=https://10.4.7.10:7443 
--kubeconfig=kubelet.kubeconfig

2. set-credentials

kubectl config set-credentials k8s-node 
--client-certificate=/opt/kubernetes/server/bin/cert/client.pem 
--client-key=/opt/kubernetes/server/bin/cert/client-key.pem 
--embed-certs=true 
--kubeconfig=kubelet.kubeconfig

3. set-context

kubectl config set-context myk8s-context 
--cluster=myk8s 
--user=k8s-node 
--kubeconfig=kubelet.kubeconfig

4. use-context

kubectl config use-context myk8s-context --kubeconfig=kubelet.kubeconfig

5. 查看生成的kubelet.kubeconfig

ll kubelet.kubeconfig

4.1.4 准备pause基础镜像

docker pull registry.cn-hangzhou.aliyuncs.com/k8s-aly/pause:3.1
docker tag registry.cn-hangzhou.aliyuncs.com/k8s-aly/pause:3.1 harbor.od.com/public/pause:3.1
docker push harbor.od.com/public/pause:3.1

4.1.5 创建kubelet启动脚本

10.4.7.21 / 10.4.7.22 / 10.4.7.23 / 10.4.7.24 操作

/opt/kubernetes/server/bin/kubelet.sh

10.4.7.21

#!/bin/sh
./kubelet 
  --anonymous-auth=false 
  --cgroup-driver systemd 
  --cluster-dns 192.168.0.2 
  --cluster-domain cluster.local 
  --runtime-cgroups=/systemd/system.slice 
  --kubelet-cgroups=/systemd/system.slice 
  --fail-swap-on="false" 
  --client-ca-file ./cert/ca.pem 
  --tls-cert-file ./cert/kubelet.pem 
  --tls-private-key-file ./cert/kubelet-key.pem 
  --hostname-override hdss7-21.host.com 
  --image-gc-high-threshold 20 
  --image-gc-low-threshold 10 
  --kubeconfig ./conf/kubelet.kubeconfig 
  --log-dir /data/logs/kubernetes/kube-kubelet 
  --pod-infra-container-image harbor.od.com/public/pause:3.1 
  --root-dir /data/kubelet

10.4.7.22

#!/bin/sh
./kubelet 
  --anonymous-auth=false 
  --cgroup-driver systemd 
  --cluster-dns 192.168.0.2 
  --cluster-domain cluster.local 
  --runtime-cgroups=/systemd/system.slice 
  --kubelet-cgroups=/systemd/system.slice 
  --fail-swap-on="false" 
  --client-ca-file ./cert/ca.pem 
  --tls-cert-file ./cert/kubelet.pem 
  --tls-private-key-file ./cert/kubelet-key.pem 
  --hostname-override hdss7-22.host.com 
  --image-gc-high-threshold 20 
  --image-gc-low-threshold 10 
  --kubeconfig ./conf/kubelet.kubeconfig 
  --log-dir /data/logs/kubernetes/kube-kubelet 
  --pod-infra-container-image harbor.od.com/public/pause:3.1 
  --root-dir /data/kubelet

10.4.7.23

#!/bin/sh
./kubelet 
  --anonymous-auth=false 
  --cgroup-driver systemd 
  --cluster-dns 192.168.0.2 
  --cluster-domain cluster.local 
  --runtime-cgroups=/systemd/system.slice 
  --kubelet-cgroups=/systemd/system.slice 
  --fail-swap-on="false" 
  --client-ca-file ./cert/ca.pem 
  --tls-cert-file ./cert/kubelet.pem 
  --tls-private-key-file ./cert/kubelet-key.pem 
  --hostname-override hdss7-23.host.com 
  --image-gc-high-threshold 20 
  --image-gc-low-threshold 10 
  --kubeconfig ./conf/kubelet.kubeconfig 
  --log-dir /data/logs/kubernetes/kube-kubelet 
  --pod-infra-container-image harbor.od.com/public/pause:3.1 
  --root-dir /data/kubelet

10.4.7.24

#!/bin/sh
./kubelet 
  --anonymous-auth=false 
  --cgroup-driver systemd 
  --cluster-dns 192.168.0.2 
  --cluster-domain cluster.local 
  --runtime-cgroups=/systemd/system.slice 
  --kubelet-cgroups=/systemd/system.slice 
  --fail-swap-on="false" 
  --client-ca-file ./cert/ca.pem 
  --tls-cert-file ./cert/kubelet.pem 
  --tls-private-key-file ./cert/kubelet-key.pem 
  --hostname-override hdss7-24.host.com 
  --image-gc-high-threshold 20 
  --image-gc-low-threshold 10 
  --kubeconfig ./conf/kubelet.kubeconfig 
  --log-dir /data/logs/kubernetes/kube-kubelet 
  --pod-infra-container-image harbor.od.com/public/pause:3.1 
  --root-dir /data/kubelet

4.1.6.授权，创建目录

10.4.7.21 / 10.4.7.22 / 10.4.7.23 / 10.4.7.24 操作

chmod +x /opt/kubernetes/server/bin/kubelet.sh 
mkdir -p /data/logs/kubernetes/kube-kubelet   /data/kubelet

4.1.7.创建supervisor配置

10.4.7.21 / 10.4.7.22 / 10.4.7.23 / 10.4.7.24 操作

/etc/supervisord.d/kube-kubelet.ini

10.4.7.21

[program:kube-kubelet-7-21]
command=/opt/kubernetes/server/bin/kubelet.sh     ; the program (relative uses PATH, can take args)
numprocs=1                                        ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin              ; directory to cwd to before exec (def no cwd)
autostart=true                                    ; start at supervisord start (default: true)
autorestart=true                                  ; retstart at unexpected quit (default: true)
startsecs=30                                      ; number of secs prog must stay running (def. 1)
startretries=3                                    ; max # of serial start failures (default 3)
exitcodes=0,2                                     ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                   ; signal used to kill process (default TERM)
stopwaitsecs=10                                   ; max num secs to wait b4 SIGKILL (default 10)
user=root                                         ; setuid to this UNIX account to run the program
redirect_stderr=true                              ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log   ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                      ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                          ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                       ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                       ; emit events on stdout writes (default false)

10.4.7.22

[program:kube-kubelet-7-22]
command=/opt/kubernetes/server/bin/kubelet.sh     ; the program (relative uses PATH, can take args)
numprocs=1                                        ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin              ; directory to cwd to before exec (def no cwd)
autostart=true                                    ; start at supervisord start (default: true)
autorestart=true                                  ; retstart at unexpected quit (default: true)
startsecs=30                                      ; number of secs prog must stay running (def. 1)
startretries=3                                    ; max # of serial start failures (default 3)
exitcodes=0,2                                     ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                   ; signal used to kill process (default TERM)
stopwaitsecs=10                                   ; max num secs to wait b4 SIGKILL (default 10)
user=root                                         ; setuid to this UNIX account to run the program
redirect_stderr=true                              ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log   ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                      ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                          ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                       ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                       ; emit events on stdout writes (default false)

10.4.7.23

[program:kube-kubelet-7-23]
command=/opt/kubernetes/server/bin/kubelet.sh     ; the program (relative uses PATH, can take args)
numprocs=1                                        ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin              ; directory to cwd to before exec (def no cwd)
autostart=true                                    ; start at supervisord start (default: true)
autorestart=true                                  ; retstart at unexpected quit (default: true)
startsecs=30                                      ; number of secs prog must stay running (def. 1)
startretries=3                                    ; max # of serial start failures (default 3)
exitcodes=0,2                                     ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                   ; signal used to kill process (default TERM)
stopwaitsecs=10                                   ; max num secs to wait b4 SIGKILL (default 10)
user=root                                         ; setuid to this UNIX account to run the program
redirect_stderr=true                              ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log   ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                      ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                          ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                       ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                       ; emit events on stdout writes (default false)

10.4.7.24

[program:kube-kubelet-7-24]
command=/opt/kubernetes/server/bin/kubelet.sh     ; the program (relative uses PATH, can take args)
numprocs=1                                        ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin              ; directory to cwd to before exec (def no cwd)
autostart=true                                    ; start at supervisord start (default: true)
autorestart=true                                  ; retstart at unexpected quit (default: true)
startsecs=30                                      ; number of secs prog must stay running (def. 1)
startretries=3                                    ; max # of serial start failures (default 3)
exitcodes=0,2                                     ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                   ; signal used to kill process (default TERM)
stopwaitsecs=10                                   ; max num secs to wait b4 SIGKILL (default 10)
user=root                                         ; setuid to this UNIX account to run the program
redirect_stderr=true                              ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log   ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                      ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                          ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                       ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                       ; emit events on stdout writes (default false)

4.1.8.启动服务并检查

10.4.7.21 / 10.4.7.22 / 10.4.7.23 / 10.4.7.24 操作

supervisorctl  update
supervisorctl status

4.1.9 检查所有节点并给节点打上标签

kubectl get nodes
kubectl label node hdss7-21.host.com node-role.kubernetes.io/master=
kubectl label node hdss7-22.host.com node-role.kubernetes.io/master=

4.1.10 为master节点打上taint

https://www.jianshu.com/p/7b147a28c828

https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

kubectl taint nodes hdss7-21.host.com hdss7-21=master:NoSchedule
kubectl taint nodes hdss7-22.host.com hdss7-22=master:NoSchedule

4.2.部署kube-proxy

主机名	角色	IP地址
hdss7-21.host.com	kube-proxy	10.4.7.21
hdss7-22.host.com	kube-proxy	10.4.7.22

4.2.1.签发kube-proxy证书

hdss7-200.host.com上

1、创建生成证书csr的json配置文件

cd /opt/certs/
vi kube-proxy-csr.json

{
    "CN": "system:kube-proxy",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

2、生成kube-proxy证书文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kube-proxy-csr.json |cfssljson -bare kube-proxy-client

4.2.2 拷贝证书文件至各节点，并创建配置

10.4.7.21 / 10.4.7.22 操作

cd /opt/kubernetes/server/bin/cert/
scp hdss7-200:/opt/certs/kube-proxy-client.pem .
scp hdss7-200:/opt/certs/kube-proxy-client-key.pem .

4.2.3 创建配置

cd /opt/kubernetes/server/bin/conf/

1. set-cluster

kubectl config set-cluster myk8s 
--certificate-authority=/opt/kubernetes/server/bin/cert/ca.pem 
--embed-certs=true 
--server=https://10.4.7.10:7443 
--kubeconfig=kube-proxy.kubeconfig

2. set-credentials

kubectl config set-credentials kube-proxy 
  --client-certificate=/opt/kubernetes/server/bin/cert/kube-proxy-client.pem 
  --client-key=/opt/kubernetes/server/bin/cert/kube-proxy-client-key.pem 
  --embed-certs=true 
  --kubeconfig=kube-proxy.kubeconfig

3. set-context

kubectl config set-context myk8s-context 
--cluster=myk8s 
--user=kube-proxy 
--kubeconfig=kube-proxy.kubeconfig

4. use-context

kubectl config use-context myk8s-context --kubeconfig=kube-proxy.kubeconfig

4.2.4.创建kube-proxy启动脚本

10.4.7.21 / 10.4.7.22 操作

1、加载ipvs模块

vi /root/ipvs.sh
#!/bin/bash
ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs"
for i in $(ls $ipvs_mods_dir|grep -o "^[^.]*")
do
  /sbin/modinfo -F filename $i &>/dev/null
  if [ $? -eq 0 ];then
    /sbin/modprobe $i
  fi
done

chmod +x /root/ipvs.sh 
sh /root/ipvs.sh 
lsmod |grep ip_vs

2、创建启动脚本

/opt/kubernetes/server/bin/kube-proxy.sh

10.7.7.21

#!/bin/sh
./kube-proxy 
  --cluster-cidr 172.7.0.0/16 
  --hostname-override hdss7-21.host.com 
  --proxy-mode=ipvs 
  --ipvs-scheduler=nq 
  --kubeconfig ./conf/kube-proxy.kubeconfig

10.7.7.22

#!/bin/sh
./kube-proxy 
  --cluster-cidr 172.7.0.0/16 
  --hostname-override hdss7-22.host.com 
  --proxy-mode=ipvs 
  --ipvs-scheduler=nq 
  --kubeconfig ./conf/kube-proxy.kubeconfig

4.2.5.授权，创建目录

chmod +x /opt/kubernetes/server/bin/kube-proxy.sh 
mkdir -p /data/logs/kubernetes/kube-proxy

4.2.6.创建supervisor配置

/etc/supervisord.d/kube-proxy.ini

10.4.7.21

[program:kube-proxy-7-21]
command=/opt/kubernetes/server/bin/kube-proxy.sh                     ; the program (relative uses PATH, can take args)
numprocs=1                                                           ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                                 ; directory to cwd to before exec (def no cwd)
autostart=true                                                       ; start at supervisord start (default: true)
autorestart=true                                                     ; retstart at unexpected quit (default: true)
startsecs=30                                                         ; number of secs prog must stay running (def. 1)
startretries=3                                                       ; max # of serial start failures (default 3)
exitcodes=0,2                                                        ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                      ; signal used to kill process (default TERM)
stopwaitsecs=10                                                      ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                            ; setuid to this UNIX account to run the program
redirect_stderr=true                                                 ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log     ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                         ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                             ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                          ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                          ; emit events on stdout writes (default false)

10.4.7.22

[program:kube-proxy-7-22]
command=/opt/kubernetes/server/bin/kube-proxy.sh                     ; the program (relative uses PATH, can take args)
numprocs=1                                                           ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                                 ; directory to cwd to before exec (def no cwd)
autostart=true                                                       ; start at supervisord start (default: true)
autorestart=true                                                     ; retstart at unexpected quit (default: true)
startsecs=30                                                         ; number of secs prog must stay running (def. 1)
startretries=3                                                       ; max # of serial start failures (default 3)
exitcodes=0,2                                                        ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                      ; signal used to kill process (default TERM)
stopwaitsecs=10                                                      ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                            ; setuid to this UNIX account to run the program
redirect_stderr=true                                                 ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log     ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                         ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                             ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                          ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                          ; emit events on stdout writes (default false)

4.2.7.启动服务并检查

supervisorctl update
supervisorctl status
yum install ipvsadm -y
ipvsadm -Ln
kubectl get svc

5.验证kubernetes集群

5.1.在任意一个节点上创建一个资源配置清单

hdss7-21.host.com上

vi /root/nginx-ds.yaml

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ds
spec:
  template:
    metadata:
      labels:
        app: nginx-ds
    spec:
      containers:
      - name: my-nginx
        image: harbor.od.com/public/nginx:1.7.9
        ports:
        - containerPort: 80
      tolerations:
      - key: hdss7-21
        operator: Exists
        effect: NoSchedule
      - key: hdss7-22
        operator: Exists
        effect: NoSchedule

5.2.应用资源配置，并检查

5.2.1.hdss7-21.host.com上

kubectl create -f /root/nginx-ds.yaml
kubectl get pods
kubectl get pods -o wide
curl -I 172.7.21.2

5.2.2.hdss7-22.host.com上

kubectl get pods
kubectl get pods -o wide
curl -I 172.7.22.2

5.2.3.查看kubernetes是否搭建好

kubectl get cs
kubectl get node
kubectl get pods

相关阅读:
java中如何创建带路径的文件
 Java 判断文件夹、文件是否存在、否则创建文件夹
 Risk Adaptive Information Flow Based Access Control
13-回顾
 Activiti操作数据库中文乱码
 12-执行流程（启动流程实例、查询任务列表、办理任务）
11-查询流程定义列表
 eclipse-jee-mars-2-win32-x86_64安装activiti
myeclipse10安装了activiti插件后创建BPMN 文件时报错，
10-部署流程定义
原文地址：https://www.cnblogs.com/cjwnb/p/12590554.html

k8s1.15.2 二进制部署

1. 实验架构

2 实验准备工作

2.1 准备虚拟机

2.2 配置基础环境

2.3 DNS服务初始化

2.3.1 安装bind

2.3.2 配置主配置文件

2.3.3 配置区域配置文件

2.3.4 配置区域数据文件

2.3.4.1 新建 区域数据文件 /var/named/host.com.zone

2.3.4.2 新建 区域数据文件 /var/named/od.com.zone

2.3.4.3 启动named服务

2.3.4.4 验证dns服务是否可用

2.3.5 设置本地dns服务（10.4.7.11）为客户端提供服务

2.3.6 配置短域名

2.3.7 配置windows 的Vmnet8网卡自己配置的DNS进行解析

2.4 准备证书签发环境

2.4.1 安装CFSSL

2.4.2 创建根证书（CA证书）的JSON配置文件

2.4.3 生成CA 证书和私钥

部署docker环境

2.5.1 安装

2.5.2 配置

2.5.3 启动

2.6 部署docker镜像仓库 harbor

2.6.1 安装docker-compose

2.6.2 下载 harbor 1.10.3

2.6.3 使用nginx反向代理 harbor.od.com

2.6.4 修改区域数据文件 /var/named/od.com.zone

2.6.5 配置harbor

2.6.6 启动harbor

2.6.7 新建项目public

2.6.8 上传镜像

3. 部署 k8s master 节点服务

3.1 部署etcd集群

3.1.1 创建基于根证书的config配置文件

3.1.2 创建etcd的请求配置文件

3.1.3 生成etcd证书和私钥

3.1.4 创建etcd用户

3.1.5 下载软件，解压，做软连接

3.1.6 创建目录，拷贝证书文件

3.1.7 拷贝生成的证书文件

3.1.9 创建etcd服务启动脚本

3.1.10 授权目录权限

3.1.11 安装supervisor软件

3.1.12 .创建supervisor配置

3.1.14 检查集群状态

3.2 部署kube-apiserver集群

3.2.1下载软件，解压，做软连接

3.2.2 签发client证书

3.2.3 签发kube-apiserver证书

3.2.4 拷贝证书和私钥

3.2.5 创建配置

3.2.6 创建apiserver启动脚本

3.2.7 授权和创建目录

3.2.8 创建supervisor配置

3.2.9 启动服务并检查

3.3.部署四层反向代理

3.3.1 配置nginx反向代理

3.3.2 配置高可用（keepalived）

3.3.2.1 check_port.sh

3.3.2.2 hdss7-11.host.com(10.4.7.11) keepalive主

3.3.2.3 hdss7-12.host.com(10.4.7.12) keepalive备

3.3.2.4 检查vip

3.3.2.5 制造一场事故

3.4.部署controller-manager

3.4.1 创建启动脚本

3.4.2 授权文件权限，创建目录

3.4.3 创建supervisor配置

3.4.4 启动服务并检查

3.5.部署kube-scheduler

3.5.1 创建启动脚本

3.5.2 授权文件权限，创建目录

3.5.3 创建supervisor配置

3.5.4 启动服务并检查

3.6 检查master节点

4. 部署Node节点服务

4.1 部署kubelet

4.1.1 签发kubelet证书

2.3.4.1 新建区域数据文件 /var/named/host.com.zone

2.3.4.2 新建区域数据文件 /var/named/od.com.zone