学习kubernetes——搭建kubernetes集群

　　学习k8s的最好方式是自己搭建一个k8s集群，并实际操作。按照官方教程，其实并不复杂，但是由于网络问题，很多软件和镜像无法下载，所以安装过程还是比较麻烦的。

　　学习k8s并不需要集群环境，个人电脑就可以搭建一个单机集群来学习。下面简单介绍下过程，会跳过比较简单的步骤，重点说下需要注意的事项

一、安装虚拟机和linux系统

　　虚拟机可以使用hyper-v，virtualbox，和vmware。我用的是VirtualBox 6.1.0版本，下载地址是https://www.virtualbox.org/wiki/Downloads。

　　系统用的是CentOS-7-x86_64-Minimal-1908。学习用的话建议使用Minimal，下载和安装都很快。下载地址是 http://isoredirect.centos.org/centos/7/isos/x86_64/，选择一个速度较快的镜像地址下载。

　　安装教程，网上很多，这里就不说了。需要建议的地方是，1、安装语言选择中文。2、软件选择最小安装，禁用KDUMP；配置网络连接

　　注意：1、虚拟机配置中设置CPU个数为2或者以上

　　　　　2、内存设置为2G以上

　　　　　3、防火墙会给k8s集群带来一些问题，这里仅为学习用，可以直接关闭防火墙。systemctl stop firewalld & systemctl disable firewalld

　　　　　4、关闭Swap。执行swapoff -a可临时关闭，编辑/etc/fstab，注释掉包含swap的那一行即可，重启后可永久关闭

　　　　　5、关闭centos图形登录界面，systemctl set-default multi-user.target

　　　　　6、添加额外的网卡，仅主机(Host-Only)网络，这样就可以从主机之间访问虚拟机

二、安装docker

　　首先参考官方文档

　　https://docs.docker.com/install/linux/docker-ce/centos/#prerequisites

　　https://kubernetes.io/docs/setup/production-environment/container-runtimes/

# Install Docker CE
## Set up the repository
### Install required packages.
yum install yum-utils device-mapper-persistent-data lvm2

### Add Docker repository.
## 注意换成阿里云地址
yum-config-manager --add-repo 
  https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

## Install Docker CE.
yum update && yum install 
  containerd.io-1.2.10 
  docker-ce-19.03.4 
  docker-ce-cli-19.03.4

## Create /etc/docker directory.
mkdir /etc/docker

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart Docker
systemctl daemon-reload
systemctl restart docker

　　设置开机启动

systemctl start docker & systemctl enable docker

　　验证安装是否成功

docker run hello-world

　　结果如下

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

三、安装Kubernetes

　　官方文档https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

　　注意：需要改成国内的镜像地址

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# Set SELinux in permissive mode (effectively disabling it)
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

systemctl enable --now kubelet

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

　　设置自启动

systemctl enable kubelet && systemctl start kubelet

四、配置K8S单机集群

　　这里使用Calico方案来部署k8s单机集群

　　官方文档：https://docs.projectcalico.org/v3.11/getting-started/kubernetes/

　　初始换环境，并下载安装k8s镜像

kubeadm init --pod-network-cidr=192.168.0.0/16

　　注意ip地址根据实际情况更改

　　由于这一步需要下载k8s的docker镜像，国内不用代理的话基本上是下载不下来的。所以这里会出错，出错的原因就是镜像pull失败。会出现类似下面的错误信息

W1229 11:23:22.589295    1688 version.go:101] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1229 11:23:22.590166    1688 version.go:102] falling back to the local client version: v1.17.0
W1229 11:23:22.590472    1688 validation.go:28] Cannot validate kube-proxy config - no validator is available
W1229 11:23:22.590492    1688 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks
        [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.17.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.17.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.17.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.17.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.4.3-0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.6.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

　　

　　执行这个解决防火墙警告

　　

firewall-cmd --permanent --add-port=6443/tcp && sudo firewall-cmd --permanent --add-port=10250/tcp && sudo firewall-cmd --reload

　　失败信息中给出了pull失败的镜像地址。github上有人已经下载所有的镜像并上传到国内地址，可以从那下载。

　　github地址：https://github.com/anjia0532/gcr.io_mirror

　　镜像地址转换规则

　　

gcr.io/namespace/image_name:image_tag 
#eq
gcr.azk8s.cn/namespace/image_name:image_tag 

# special
k8s.gcr.io/{image}/{tag} <==> gcr.io/google-containers/{image}/{tag} <==> gcr.azk8s.cn/namespace/image_name:image_tag 

　　例如上面初始话下载失败的镜像地址是k8s.gcr.io/kube-apiserver:v1.17.0。

k8s.gcr.io/kube-apiserver:v1.17.0
转换为
gcr.azk8s.cn/google-containers/kube-apiserver:v1.17.0

　　下载镜像

docker pull gcr.azk8s.cn/google-containers/kube-controller-manager:v1.17.0
docker pull gcr.azk8s.cn/google-containers/kube-scheduler:v1.17.0
docker pull gcr.azk8s.cn/google-containers/kube-proxy:v1.17.0
docker pull gcr.azk8s.cn/google-containers/pause:3.1
docker pull gcr.azk8s.cn/google-containers/etcd:3.4.3-0
docker pull gcr.azk8s.cn/google-containers/coredns:1.6.5

　　由于，kubeadm init拉取的镜像地址是官方的地址，因此我们需要打对应的tag

docker tag gcr.azk8s.cn/google-containers/kube-apiserver:v1.17.0 k8s.gcr.io/kube-apiserver:v1.17.0
docker tag gcr.azk8s.cn/google-containers/kube-controller-manager:v1.17.0 k8s.gcr.io/kube-controller-manager:v1.17.0
docker tag gcr.azk8s.cn/google-containers/kube-scheduler:v1.17.0 k8s.gcr.io/kube-scheduler:v1.17.0
docker tag gcr.azk8s.cn/google-containers/kube-proxy:v1.17.0 k8s.gcr.io/kube-proxy:v1.17.0
docker tag gcr.azk8s.cn/google-containers/pause:3.1 k8s.gcr.io/pause:3.1
docker tag gcr.azk8s.cn/google-containers/etcd:3.4.3-0 k8s.gcr.io/etcd:3.4.3-0
docker tag gcr.azk8s.cn/google-containers/coredns:1.6.5 k8s.gcr.io/coredns:1.6.5

　　同样的方式完成所有镜像的下载。

　　如果有多台服务器可以进行同样的操作。

　　设置主机域名

　　编辑/etc/hostname，将hostname修改为k8s-node1
　　编辑/etc/hosts，追加内容 IP k8s-node1

　　然后在主节点再次执行初始化，如果失败，可以先执行kubeadm reset

kubeadm init --pod-network-cidr=192.168.0.0/16

## 如果需要多个节点
kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=192.168.56.104

　　继续执行

　　

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

　　安装Calico

kubectl apply -f https://docs.projectcalico.org/v3.11/manifests/calico.yaml

　　这一步也需要下载相应的calico镜像，也可能下载失败，大家可以去上面的yaml文件中，找到需要的镜像，然后搜索下载方式，这里就不说了。

　　验证是否成功

　　

watch kubectl get pods --all-namespaces

　　结果如下就表示成功了

　　

NAMESPACE    NAME                                       READY  STATUS   RESTARTS  AGE
kube-system  calico-kube-controllers-6ff88bf6d4-tgtzb   1/1    Running  0         2m45s
kube-system  calico-node-24h85                          1/1    Running  0         2m43s
kube-system  coredns-846jhw23g9-9af73                   1/1    Running  0         4m5s
kube-system  coredns-846jhw23g9-hmswk                   1/1    Running  0         4m5s
kube-system  etcd-jbaker-1                              1/1    Running  0         6m22s
kube-system  kube-apiserver-jbaker-1                    1/1    Running  0         6m12s
kube-system  kube-controller-manager-jbaker-1           1/1    Running  0         6m16s
kube-system  kube-proxy-8fzp2                           1/1    Running  0         5m16s
kube-system  kube-scheduler-jbaker-1                    1/1    Running  0         5m41s

　　如果calico-node出现ErrorImagePull等状态，就表示这个镜像没有下载成功，需要自己手动去国内镜像地址去下载，名称和版本号见https://docs.projectcalico.org/v3.11/manifests/calico.yaml

　　配置master节点为work节点

kubectl taint nodes --all node-role.kubernetes.io/master-

　　结果如下

node/<your-hostname> untainted

　　最后运行

　　

kubectl get nodes -o wide

　　出现类型如下结果就表示成功了

NAME              STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
<your-hostname>   Ready    master   52m   v1.12.2   10.128.0.28   <none>        Ubuntu 18.04.1 LTS   4.15.0-1023-gcp   docker://18.6.1