老男孩K8S集群部署（二）

参考https://blog.stanley.wang/

本文是我学习了老男孩k8s后做的笔记，整理成博客发表出来便于自己之后复习回顾。

部署etcd集群

部署etcd前也要给etcd创建证书。

我们这里是给hdss172-22.host.com，hdss172-23.host.com，hdss172-24.host.com三台安装etcd

主机名	角色	ip
hdss172-22.host.com	etcd leader	192.168.172.22
hdss172-23.host.com	etcd follow	192.168.172.23
hdss172-24.host.com	etcd follow	192.168.172.23

创建生成证书签名请求（csr）的JSON配置文件

运维主机hdss172-25.host.com上：

hosts字段的含义是etcd有可能部署在哪些主机上，有可能的都要写出来。否则通信会出错。不支持网段，只支持单个ip。

cat /opt/certs/etcd-peer-csr.json
{
    "CN": "etcd-peer",
    "hosts": [
        "192.168.172.21",
        "192.168.172.22",
        "192.168.172.23",
        "192.168.172.24",
        "192.168.172.25"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

peer段，互相通信。

vim ca-config.json
{
    "signing": {
        "default": {
            "expiry": "175200h"
        },
        "profiles": {
            "server": {
                 "expiry": "175200h",
                 "usages": [
                     "signing", 
                     "key encipherment",
                     "server auth"
            ]
        },
             "client": {
                 "expiry": "175200h",
                 "usages": [
                      "signing",
                      "key encipherment",
                      "server auth"
             ]
        },
             "peer": {
                  "expiry": "175200h",
                  "usages": [
                       "signing",
                       "key encipherment",                  
                       "server auth",
                       "client auth"
              ]
         }
    }  
 }

}

签发证书：

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json

加入管道：

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json | cfssl-json -bare etcd-peer

这里以hdss172-22.host.com主机的部署为例，另外两台主机部署方法类似。

部署etcd

创建用户：

-M 不要家目录
useradd -s /sbin/nologin -M etcd   

下载软件：
下载地址：github.com/etcd-io/etcd/releases。建议使用3.1.x的版本，比较稳定。

cd /opt/src
tar zxf etcd-v3.1.8-linux-amd64.tar.gz -C /opt/
mv etcd-v3.1.8-linux-amd64/ etcd-v3.1.8
ln -s /opt/etcd-v3.1.8 /opt/etcd

创建目录：

mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server

拷贝证书：
启动etcd需要3个证书
将运维主机192.168.172.25上生成的ca.pem、etcd-peer-key.pem、etcd-peer.pem拷贝到/opt/etcd/certs目录中，注意私钥文件权限600。

cd /opt/etcd/certs
scp 192.168.172.25:/opt/certs/ca.pem .
scp 192.168.172.25:/opt/certs/etcd-peer-key.pem .
scp 192.168.172.25:/opt/certs/etcd-peer.pem . 

创建etcd服务启动脚本:

hdss172-22.host.com上：

vim /opt/etcd/etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-172-22 
       --data-dir /data/etcd/etcd-server 
       --listen-peer-urls https://192.168.172.22:2380 
       --listen-client-urls https://192.168.172.22:2379,http://127.0.0.1:2379 
       --quota-backend-bytes 8000000000 
       --initial-advertise-peer-urls https://192.168.172.22:2380 
       --advertise-client-urls https://192.168.172.22:2379,http://127.0.0.1:2379 
       --initial-cluster  etcd-server-172-22=https://192.168.172.22:2380,etcd-server-172-23=https://192.168.172.23:2380,etcd-server-172-24=https://192.168.172.24:2380 
       --ca-file ./certs/ca.pem 
       --cert-file ./certs/etcd-peer.pem 
       --key-file ./certs/etcd-peer-key.pem 
       --client-cert-auth  
       --trusted-ca-file ./certs/ca.pem 
       --peer-ca-file ./certs/ca.pem 
       --peer-cert-file ./certs/etcd-peer.pem 
       --peer-key-file ./certs/etcd-peer-key.pem 
       --peer-client-cert-auth 
       --peer-trusted-ca-file ./certs/ca.pem 
       --log-output stdout

chmod +x /opt/etcd/etcd-server-startup.sh
chown -R etcd.etcd /opt/etcd-v3.1.8/
chown -R etcd.etcd /data/etcd/
chown -R etcd.etcd /data/logs/etcd-server/

使用supervisor启动:

yum install supervisor.noarch –y
systemctl start supervisord.service
systemctl enable supervisord.service

创建supervisor的启动文件:

vim /etc/supervisord.d/etcd-server.ini
[program:etcd-server-172-22]
command=/opt/etcd/etcd-server-startup.sh                        ; the program (relative uses PATH, can take args)
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/opt/etcd                                             ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=22                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd                                                       ; setuid to this UNIX account to run the program
redirect_stderr=false                                           ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log           ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)
stderr_logfile=/data/logs/etcd-server/etcd.stderr.log           ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=4                                        ; # of stderr logfile backups (default 10)
stderr_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stderr_events_enabled=false                                     ; emit events on stderr writes (default false)

注意：etcd集群各主机启动配置略有不同，配置其他节点时注意修改。

supervisorctl update
supervisorctl status
tail -fn 200 /data/logs/etcd-server/etcd.stdout.log
netstat -antlp | grep etcd

所以，etcd启动了。

以上只是用1台主机举例，其他两台主机也需要做同样的操作。
部署完成后，在任意一台主机上检测etcd集群的状态。

检测办法一：

./etcdctl cluster-health

检测办法二：

./etcdctl member list

22主机是leader

安装部署主控节点服务

部署kube-apiserver集群

集群规划

主机名	角色	ip
hdss172-21.host.com	4层负载均衡	192.168.172.21
hdss172-22.host.com	4层负载均衡	192.168.172.22
hdss172-23.host.com	kube-apiserver	192.168.172.23
hdss172-24.host.com	kube-apiserver	192.168.172.24

注意：这里192.168.172.21和192.168.172.22使用nginx做4层负载均衡器，用keepalived跑一个vip：192.168.172.100，代理两个kube-apiserver，实现高可用
这里部署文档以hdss172-23.host.com主机为例，另外一台运算节点安装部署方法类似
Github链接：https://github.com/kubernetes/kubernetes

下载这个

演示安装部署1.15.2

kubernetes-server-linux-amd64-v1.15.2.tar.gz压缩包下载到/opt/src下
tar zxf kubernetes-server-linux-amd64-v1.15.2.tar.gz -C /opt/
mv kubernetes/ kubernetes-v1.15.2
ln -s kubernetes-v1.15.2/ kubernetes

cd kubernetes
rm -f kubernetes-src.tar.gz
删除go语言编写的源码包

.tar 文件 是docker镜像
rm -f *.tar
rm -f *_tag

部署apiserver的第一件事，签发client证书。是apiserver和etcd集群通信时用的证书。

etcd是server端，apiserver是客户端

在hdss172-25.host.com上:

vim /opt/certs/client-csr.json
{
    "CN": "k8s-node",
    "hosts": [
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

生成client证书：

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json | cfssl-json -bare client

生成server证书：

hosts字段表示apiserver可能使用的ip地址

vim /opt/certs/apiserver-csr.json
{
    "CN": "apiserver",
    "hosts": [
        "127.0.0.1",
        "10.4.0.1",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local",
        "192.168.172.100",
        "192.168.172.21",
        "192.168.172.22",
        "192.168.172.23",
        "192.168.172.24"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

生成server证书：

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json | cfssl-json -bare apiserver

在hdss172-23.host.com主机上：

mkdir cert

拷贝证书：apiserver-key.pem，apiserver.pem，ca-key.pem，ca.pem，client-key.pem，client.pem

scp 192.168.172.25:/opt/certs/apiserver-key.pem .
scp 192.168.172.25:/opt/certs/apiserver.pem .
scp 192.168.172.25:/opt/certs/ca-key.pem .
scp 192.168.172.25:/opt/certs/ca.pem .
scp 192.168.172.25:/opt/certs/client-key.pem .
scp 192.168.172.25:/opt/certs/client.pem .

创建apiserver启动的配置文件（资源清单）:

mkdir conf
vim /opt/kubernetes/server/bin/conf/audit.yaml
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]

  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"

  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"

创建apiserver的启动脚本：

vim /opt/kubernetes/server/bin/kube-apiserver.sh
#!/bin/bash
./kube-apiserver 
  --apiserver-count 2 
  --audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log 
  --audit-policy-file ./conf/audit.yaml 
  --authorization-mode RBAC 
  --client-ca-file ./cert/ca.pem 
  --requestheader-client-ca-file ./cert/ca.pem 
  --enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota 
  --etcd-cafile ./cert/ca.pem 
  --etcd-certfile ./cert/client.pem 
  --etcd-keyfile ./cert/client-key.pem 
  --etcd-servers https://192.168.172.22:2379,https://192.168.172.23:2379,https://192.168.172.24:2379 
  --service-account-key-file ./cert/ca-key.pem 
  --service-cluster-ip-range 10.4.0.1/16 
  --service-node-port-range 3000-29999 
  --target-ram-mb=1024 
  --kubelet-client-certificate ./cert/client.pem 
  --kubelet-client-key ./cert/client-key.pem 
  --log-dir  /data/logs/kubernetes/kube-apiserver 
  --tls-cert-file ./cert/apiserver.pem 
  --tls-private-key-file ./cert/apiserver-key.pem 
  --v 2

chmod +x kube-apiserver.sh

查看帮助

./kube-apiserver –help

创建脚本中的日志路径

mkdir -p /data/logs/kubernetes/kube-apiserver/audit-log

编辑supervisor文件：

vim /etc/supervisord.d/kube-apiserver.ini
[program:kube-apiserver]
command=/opt/kubernetes/server/bin/kube-apiserver.sh            ; the program (relative uses PATH, can take args)
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                            ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=22                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                       ; setuid to this UNIX account to run the program
redirect_stderr=false                                           ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log        ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)
stderr_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stderr.log        ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=4                                        ; # of stderr logfile backups (default 10)
stderr_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stderr_events_enabled=false                                     ; emit events on stderr writes (default false)

启动kube-apiserver：

supervisorctl update
supervisorctl status