vmware快速搭建k8s集群 (vmware16|centos8|docker19.03.15|k8s1.16.9|calico3.16)(kubeadm安装方式)

1.安装docker （所有机器）

1.设置国内yum源
rm -rf /etc/yum.repos.d/*
wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-vault-8.5.2111.repo
2.安装docker
dnf update
dnf install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
dnf list docker-ce --showduplicates
dnf install docker-ce-19.03.15
3.启动docker
systemctl enable docker & systemctl start docker
4.设置docker镜像加速器
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": [
            "https://oemgr772.mirror.aliyuncs.com",
            "https://docker.mirrors.ustc.edu.cn"
  ]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

centos8问题

CentOS8安装docker报错：problem with installed package podman-1.6.4-10.module_el8.2.0+305+5e198a41.x86_64
‘centos8默认使用podman代替docker
解决：yum install --allowerasing docker-ce

2.k8s环境准备 （所有机器）

1.关闭防火墙
systemctl stop firewalld & systemctl disable firewalld
2.关闭selinux
# 临时禁用selinux
setenforce 0
# 永久关闭 修改/etc/sysconfig/selinux文件设置
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
3.禁用交换分区
swapoff -a
# 永久禁用，打开/etc/fstab注释掉swap那一行。
sed -i 's/.*swap.*/#&/' /etc/fstab
4.修改内核参数
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

1.设置网卡静态IP
# vi /etc/sysconfig/network-scripts/ifcfg-ens33
GATEWAY=192.168.239.2
IPADDR=192.168.239.138
NETMASK=255.255.255.0
DNS1=192.168.239.2
DNS2=1.2.4.8

重新载入一下配置文件
nmcli c reload

重启网卡
nmcli c up ens33 或 nmcli d reapply ens33 或 nmcli d connect ens33

scp /etc/sysconfig/network-scripts/ifcfg-ens33 192.168.239.141:/etc/sysconfig/network-scripts/
scp /etc/sysconfig/network-scripts/ifcfg-ens33 192.168.239.142:/etc/sysconfig/network-scripts/


2.设置时区
# timedatectl set-timezone Asia/Shanghai 

3.设置主机名
# hostnamectl set-hostname k8s-master 
# hostnamectl set-hostname k8s-node01 
# hostnamectl set-hostname k8s-node02

4.配置域名解析
vim /etc/hosts
192.168.239.138 k8s-master
192.168.239.139 k8s-node01
192.168.239.140 k8s-node02

scp /etc/hosts k8s-node01:/etc/
scp /etc/hosts k8s-node02:/etc/

重复步骤3

yum源设置epel,kubernetes 镜像  (略)

注意： 记得把kubernetes.repo里面gpgcheck、repo_gpgcheck俩值改成0

3.安装k8s

k8s-master安装kubectl、kubeadm、kubelet

列出所有可更新的软件清单命令： 
dnf check-update

安装kubectl、kubeadm、kubelet
dnf install -y kubectl-1.16.9-0 kubeadm-1.16.9-0 kubelet-1.16.9-0
systemctl enable kubelet && systemctl start kubelet

查看kubelet实时日志
# journalctl -xefu kubelet 

k8s-master初始化

# 初始化容器
kubeadm init  \
 --kubernetes-version=1.16.9  \
 --apiserver-advertise-address=192.168.239.138   \
 --image-repository registry.aliyuncs.com/google_containers \
 --service-cidr=10.10.0.0/16 \
 --pod-network-cidr=10.122.0.0/16

日志

【
[init] Using Kubernetes version: v1.16.9
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING FileExisting-tc]: tc not found in system path
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.15. Latest validated version: 18.09
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.10.0.1 192.168.239.138]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.239.138 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.239.138 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 16.002454 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: vcxm7y.7fe8g0ma5v9gmj38
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.239.138:6443 --token vcxm7y.7fe8g0ma5v9gmj38 \
    --discovery-token-ca-cert-hash sha256:904c6e63b1a7f9297a7897e3676fa941e32aa04e9dc1e7d8dedb4808abe8623f
】

按日志提示继续执行

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

k8s-node01、k8s-node02安装kubeadm、kubelet

安装kubeadm、kubelet
dnf install -y kubeadm-1.16.9-0 kubelet-1.16.9-0
systemctl enable kubelet && systemctl start kubelet
查看kubelet实时日志
journalctl -xefu kubelet 

k8s-node01、k8s-node02加入集群 

上面日志有哦

kubeadm join 192.168.239.138:6443 --token xx 

k8s-master查看

查看命名空间
kubectl get ns 
查看命名空间下的容器
kubectl get pods -n kube-system
查看容器信息
kubectl describe pods xxx -n kube-system   

4.安装网络插件calico

容器网络接口（Container Network Interface），简称 CNI。该接口只有四个方法，添加网络、删除网络、添加网络列表、删除网络列表。

https://jimmysong.io/kubernetes-handbook/concepts/cni.html

https://help.aliyun.com/document_detail/86745.html

# mkdir -p /app/kube/yaml && cd /app/kube/yaml
# curl -O https://docs.projectcalico.org/v3.16/manifests/calico.yaml

修改calico.yaml文件

1). calico/node:v3.16.10》env：

            - name: KUBERNETES_SERVICE_HOST
              value: "192.168.239.138" 
            - name: KUBERNETES_SERVICE_PORT
              value: "6443"  
            - name: KUBERNETES_SERVICE_PORT_HTTPS
              value: "6443"  

2). calico/kube-controllers:v3.16.10 ：

      volumes:
       - name: test-volume
         hostPath:
           path: /root/.kube
      containers:
        - name: calico-kube-controllers
          image: calico/kube-controllers:v3.16.10
          volumeMounts:
            - name: test-volume 
              mountPath: /test-pd
          env:
            # Choose which controllers to run.
            - name: ENABLED_CONTROLLERS
              value: node
            - name: DATASTORE_TYPE
              value: kubernetes
          readinessProbe:
            exec:
              command:
              - /usr/bin/check-status
              - -r

 部署calico

# grep 3.16 /app/kube/calico.yaml    //查看镜像
# kubectl create -f calico.yaml       //创建Calico Pod
# kubectl apply -f calico.yaml        //更新Calico Pod

 5.安装dashboard(用火狐浏览器)

k8s官网文档：https://kubernetes.io/zh/docs/tasks/access-application-cluster/web-ui-dashboard/
git地址         ：https://github.com/kubernetes/dashboard/releases

###kubernetes安装dashboard步骤

k8s官网文档：https://kubernetes.io/zh/docs/tasks/access-application-cluster/web-ui-dashboard/
git地址 ：https://github.com/kubernetes/dashboard/releases

下载:
wget https://kuboard.cn/install-script/k8s-dashboard/v2.0.0-beta5.yaml

修改：
vim recommended.yaml
1.获得管理员权限
将clusterrolebinding附近
name: kubernetes-dashboard
改成
name: cluster-admin
2.固定svc端口
type: NodePort
nodePort: 30000
3. 让dashboard容器部署在主节点
方式一：设置nodeName
nodeName: k8s-master
方式二：设置nodeSelector
#创建标签
kubectl label node k8s-master type=master
#标签选择器
spec:
nodeSelecor:
type:master

4.启动
kubectl create -f recommended.yaml //创建执行
kubectl apply -f recommended.yaml //修改后重新执行

5.查看
kubectl get svc,po -A

kubectl get ns
kubectl get pod -A | grep dashboard
kubectl get service -A | grep dashboard
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Status}}"

kubectl cluster-info //集群是否可以访问
kubectl api-resources | grep ClusterRole //查看资源列表

6.访问
方式1：kubectl proxy --address='0.0.0.0' --accept-hosts='^*$' --port=8001
http://192.168.239.138:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

方式2：NodePort

lsof -i tcp:8001

http://192.168.239.138:8001/api/v1/namespaces/kubernetes-dashboard/http:10.10.31.54:8443/kubernetes-dashboard

获取登录需要的token
kubectl get secrets -n kube-system
kubectl get secrets -n kubernetes-dashboard
kubectl describe secrets kubernetes-dashboard-token-6gpmq -n kubernetes-dashboard

https://192.168.239.138:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

卸载kubernetes-dashboard 3种：
方法1：（效果不干净）
 kubectl delete ns kubernetes-dashboard 
方法2：（效果不错）
 kubectl delete -f recommended-2.2.0.yaml 
方法3：（效果不错）
1). 删除pod
 kubectl get pods -A | grep dashboard
 kubectl delete deployment kubernetes-dashboard      -n kubernetes-dashboard
 kubectl delete deployment dashboard-metrics-scraper -n kubernetes-dashboard
2). 删除service
 kubectl get service -A | grep dashboard
 kubectl delete service kubernetes-dashboard       -n kubernetes-dashboard
 kubectl delete service dashboard-metrics-scraper  -n kubernetes-dashboard
3). 删除账户和密钥 
kubectl delete sa     kubernetes-dashboard            -n kubernetes-dashboard
kubectl delete secret kubernetes-dashboard-certs      -n kubernetes-dashboard
kubectl delete secret kubernetes-dashboard-key-holder -n kubernetes-dashboard

知识：
什么是Service?
https://www.cnblogs.com/infodriven/p/16257871.html

设置service的nodeport以后外部无法访问对应的端口的问题？
https://blog.51cto.com/u_11288550/2378289

ClusterIP, NodePorts, LoadBalancers, Ingress的区别及配置？
https://zhuanlan.zhihu.com/p/65825116

Kubernetes的三种外部访问方式：NodePort、LoadBalancer 和 Ingress？
http://dockone.io/article/4884

kubernetes入门之kube-proxy实现原理 ?
https://www.cnblogs.com/xuxinkun/p/5799986.html

k8s部署dashboard与踩坑?
https://pj1987111.github.io/posts/k8s/k8s%E9%83%A8%E7%BD%B2dashboard%E4%B8%8E%E8%B8%A9%E5%9D%91/

博客：k8s部署dashboard与踩坑
https://pj1987111.github.io/posts/k8s/k8s%E9%83%A8%E7%BD%B2dashboard%E4%B8%8E%E8%B8%A9%E5%9D%91/