kubernetes安装： master,etcd: node: 前提：基于主机名通信； 时间同步； 关闭firewalld和iptables.service OS:centos7.3，extra 步骤： etcd cluster，仅master节点； flannel，集群的所有节点； 配置k8s的master：仅master节点：kubernetes-master 启动的服务：kube-apiserver,kube-scheduler,kube-controller-manager 配置的K8s的node节点：kubernetes-node 先设定启动docker服务； 启动的k8s服务：kube-proxy,kubelet kubeadm: 1、master,nodes安装kubelet,kubeadm,docker 2、master:kubeadm init 3、nodes:kubeadm join https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.10.md 关闭firewalld,selinux 1、配置yum源： docker yum源： wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo #阿里云yum源： wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo yum clean all yum makecache #docker yum源 cat >> /etc/yum.repos.d/docker.repo <<EOF [docker-repo] name=Docker Repository baseurl=http://mirrors.aliyun.com/docker-engine/yum/repo/main/centos/7 enabled=1 gpgcheck=0 EOF kubernertes yum源： cat >> /etc/yum.repos.d/k8s.repo <<EOF [k8s] name=k8s baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ gpgcheck=0 enabled=1 EOF gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg enabled=1 若要使用key: # wget https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg # yum --import yum-key.gpg 其它yum源： wget http://mirrors.aliyun.com/repo/Centos-7.repo 2、master端操作： # yum install -y docker-ce kubelet kubeadm kubectl # rpm -ql docker-ce /usr/bin/docker-init /usr/bin/docker-proxy /usr/bin/dockerd-ce /usr/lib/systemd/system/docker.service /usr/lib/systemd/system/docker.socket /var/lib/docker-engine/distribution_based_engine-ce.json # rpm -ql kubelet /etc/kubernetes/manifests /etc/sysconfig/kubelet /usr/bin/kubelet /usr/lib/systemd/system/kubelet.service # rpm -ql kubeadm /usr/bin/kubeadm /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf # rpm -ql kubectl /usr/bin/kubectl # vi /usr/lib/systemd/system/docker.service Environment="HTTPS_PROXY=http://www.ik8s.io:10080 Environment=""NO_PROXY=127.0.0.0.8,192.168.31.0/16" # systemctl daemon-reload # systemctl start docker.service # cat /proc/sys/net/bridge/bridge-nf-call-iptables 1 # systemctl enable docker.service # systemctl enable kubelet # ss -tnl # vi /etc/sysconfig/kubelet #禁用swap KUBELET_EXTRA_ARGS="--fail-swap-on=false" # kubeadm init --help # kubeadm init --kubernetes-version=stable-1.11 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap # kubeadm init --kubernetes-version=stable-1 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap --kubernetes-version=stable-1.11 #指定kubernetes版本 --pod-network-cidr=10.244.0.0/16 #指定pod的网段 --service-cidr=10.96.0.0/12 #指定service的网段 --ignore-preflight-errors=Swap #忽略swap 初始化失败，国内网站无法访问dl.k8s.io/，因此需要事先把这些镜像拉取下来： could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 方法1：有个墙外的代理服务器，对docker配置代理，需修改/etc/sysconfig/docker文件，添加： HTTP_PROXY=http://proxy_ip:port http_proxy=$HTTP_PROXY 重启docker：systemctl restart docker # docker image ls # kubeadm config images pull # kubeadm config images list #初始化时需要的镜像 k8s.gcr.io/kube-apiserver:v1.14.0 k8s.gcr.io/kube-controller-manager:v1.14.0 k8s.gcr.io/kube-scheduler:v1.14.0 k8s.gcr.io/kube-proxy:v1.14.0 k8s.gcr.io/pause:3.1 k8s.gcr.io/etcd:3.3.10 k8s.gcr.io/coredns:1.3.1 方法2：通过 docker.io/mirrorgooglecontainers中转一下https://hub.docker.com/u/mirrorgooglecontainers # kubeadm config images list |sed -e 's/^/docker pull /g' -e 's#k8s.gcr.io#docker.io/mirrorgooglecontainers#g' |sh -x #下载需要的镜像 # docker images |grep mirrorgooglecontainers |awk '{print "docker tag ",$1":"$2,$1":"$2}' |sed -e 's#mirrorgooglecontainers#k8s.gcr.io#2' |sh -x #重命名镜像 # docker images |grep mirrorgooglecontainers |awk '{print "docker rmi ", $1":"$2}' |sh -x #删除mirrorgooglecontainers镜像 # docker pull docker.io/mirrorgooglecontainers/kube-apiserver:v1.14.0 # docker pull docker.io/mirrorgooglecontainers/kube-controller-manager:v1.14.0 # docker pull docker.io/mirrorgooglecontainers/kube-scheduler:v1.14.0 # docker pull docker.io/mirrorgooglecontainers/kube-proxy:v1.14.0 # docker pull docker.io/mirrorgooglecontainers/pause:3.1 # docker pull docker.io/mirrorgooglecontainers/etcd:3.3.10 # docker tag mirrorgooglecontainers/kube-apiserver:v1.14.0 k8s.gcr.io/kube-apiserver:v1.14.0 # docker tag mirrorgooglecontainers/kube-proxy:v1.14.0 k8s.gcr.io/kube-proxy:v1.14.0 # docker tag mirrorgooglecontainers/kube-controller-manager:v1.14.0 k8s.gcr.io/kube-controller-manager:v1.14.0 # docker tag mirrorgooglecontainers/kube-scheduler:v1.14.0 k8s.gcr.io/kube-scheduler:v1.14.0 # docker tag mirrorgooglecontainers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10 # docker tag mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1 coredns没包含在docker.io/mirrorgooglecontainers中，需要手工从coredns官方镜像转换下。 # docker pull coredns/coredns:1.3.1 # docker tag coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1 # docker rmi coredns/coredns:1.3.1 master初始化: # kubeadm init --kubernetes-version=stable-1 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap # kubectl get -h # kubectl get cs #kubectl get componentstatus NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true"} 或： # kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=NumCPU 因为后面要安装网络插件flannel ，所有这里要添加参数， --pod-network-cidr=10.244.0.0/16，10.244.0.0/16是flannel插件固定使用的ip段，它的值取决于你准备安装哪个网络插件 如果要自定义配置，先kubeadm config print init-defaults >kubeadm.conf，再修改，改完指定配置文件路径--config /root/kubeadm.conf 指定Kubenetes版本--kubernetes-version，如果不指定该参数，会从google网站下载最新的版本信息，因为它的默认值是stable-1。 因为使用的是虚拟机，只分配一个cpu，所以指定了参数--ignore-preflight-errors=NumCPU，如果你的cpu足够，不要添加这个参数. 初始化参数说明： -apiserver-advertise-address string API Server将要广播的监听地址。如指定为 `0.0.0.0` 将使用缺省的网卡地址。 --apiserver-bind-port int32 缺省值: 6443 API Server绑定的端口 --apiserver-cert-extra-sans stringSlice 可选的额外提供的证书主题别名（SANs）用于指定API Server的服务器证书。可以是IP地址也可以是DNS名称。 --cert-dir string 缺省值: "/etc/kubernetes/pki" 证书的存储路径。 --config string kubeadm配置文件的路径。警告：配置文件的功能是实验性的。 --cri-socket string 缺省值: "/var/run/dockershim.sock" 指明要连接的CRI socket文件 --dry-run 不会应用任何改变；只会输出将要执行的操作。 --feature-gates string 键值对的集合，用来控制各种功能的开关。可选项有: Auditing=true|false (当前为ALPHA状态 - 缺省值=false) CoreDNS=true|false (缺省值=true) -h, --help 获取init命令的帮助信息 --ignore-preflight-errors stringSlice 忽视检查项错误列表，列表中的每一个检查项如发生错误将被展示输出为警告，而非错误。 例如: 'IsPrivilegedUser,Swap'. 如填写为 'all' 则将忽视所有的检查项错误。 --kubernetes-version string 缺省值: "stable-1" 为control plane选择一个特定的Kubernetes版本。 --node-name string 指定节点的名称。 --pod-network-cidr string 指明pod网络可以使用的IP地址段。 如果设置了这个参数，control plane将会为每一个节点自动分配CIDRs。 --service-cidr string 缺省值: "10.96.0.0/12" 为service的虚拟IP地址另外指定IP地址段 --service-dns-domain string 缺省值: "cluster.local" 为services另外指定域名, 例如： "myorg.internal". --skip-token-print 不打印出由 `kubeadm init` 命令生成的默认令牌。 --token string 这个令牌用于建立主从节点间的双向受信链接。格式为 [a-z0-9]{6}\.[a-z0-9]{16} - 示例： abcdef.0123456789abcdef --token-ttl duration 缺省值: 24h0m0s 令牌被自动删除前的可用时长 (示例： 1s, 2m, 3h). 如果设置为 '0', 令牌将永不过期。 ----------------------- 部署pod网络插件:flannel插件 选择flannel作为网络插件： vim /etc/sysctl.conf，添加以下内容 net.ipv4.ip_forward=1 net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 修改后，及时生效 sysctl -p 地址：https://github.com/coreos/flannel # kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml # wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml flannel 默认会使用主机的第一张网卡，如果你有多张网卡，需要通过配置单独指定。修改 kube-flannel.yml 中的以下部分 vim kube-flannel.yml containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.10.0-amd64 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr - --iface=ens33 #添加 # kubectl apply -f kube-flannel.yml 查看各组件的状态： # kubectl get cs # kubectl get componentstatus # kubectl get nodes NAME STATUS ROLES AGE VERSION vm1.cluster.com Ready master 13m v1.14.0 # kubectl get pod No resources found. # kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-fb8b8dccf-4sr5b 1/1 Running 0 14m coredns-fb8b8dccf-rmj7h 1/1 Running 0 14m etcd-vm1.cluster.com 1/1 Running 0 13m kube-apiserver-vm1.cluster.com 1/1 Running 0 13m kube-controller-manager-vm1.cluster.com 1/1 Running 0 13m kube-flannel-ds-amd64-rnght 1/1 Running 0 2m30s kube-proxy-mxjwr 1/1 Running 0 14m kube-scheduler-vm1.cluster.com 1/1 Running 0 13m # kubectl get ns #名称空间 NAME STATUS AGE default Active 16m kube-node-lease Active 16m kube-public Active 16m kube-system Active 16m 3、在node节点上操作： # yum install -y docker-ce kubelet kubeadm # vi /usr/lib/systemd/system/docker.service Environment="HTTPS_PROXY=http://www.ik8s.io:10080 Environment=""NO_PROXY=127.0.0.0.8,192.168.31.0/16" # vi /etc/sysconfig/kubelet #禁用swap KUBELET_EXTRA_ARGS="--fail-swap-on=false" # systemctl start docker # systemctl enable docker # systemctl enable kubelet 注意，这里不需要启动kubelet，初始化的过程中会自动启动的，如果此时启动了会出现如下报错，忽略即可。日志在tail -f /var/log/messages failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file “/var/lib/kubelet/config.yaml”, error: open /var/lib/kubelet/config.yaml: no such file or directory kubeadm join 192.168.31.11:6443 --token rquyna.2jykkhlqq7zr306v \ --discovery-token-ca-cert-hash sha256:f7d07c0ba9ce136a0fb5d3a623146c51e17dfe49d69273474dc4ac902415dc79 --ignore-preflight-errors=Swap node节点所需要的几个镜像： k8s.gcr.io/kube-proxy-amd64:v1.10.0 k8s.gcr.io/pause-amd64:3.1 quay.io/coreos/flannel:v0.9.1-amd64（为网络插件的镜像，这里选择flannel为网络插件） # docker pull docker.io/mirrorgooglecontainers/kube-proxy:v1.14.0 # docker pull docker.io/mirrorgooglecontainers/pause:3.1 # docker tag mirrorgooglecontainers/kube-proxy:v1.14.0 k8s.gcr.io/kube-proxy:v1.14.0 # docker tag mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1 # docker pull coredns/coredns:1.3.1 # docker tag coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1 # docker rmi coredns/coredns:1.3.1 node节点上会拉取如下镜像： # docker images REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.14.0 5cd54e388aba 10 days ago 82.1MB k8s.gcr.io/kube-scheduler v1.14.0 00638a24688b 10 days ago 81.6MB k8s.gcr.io/kube-apiserver v1.14.0 ecf910f40d6e 10 days ago 210MB k8s.gcr.io/kube-controller-manager v1.14.0 b95b1efa0436 10 days ago 158MB quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 2 months ago 52.6MB coredns/coredns 1.3.1 eb516548c180 2 months ago 40.3MB k8s.gcr.io/coredns 1.3.1 eb516548c180 2 months ago 40.3MB k8s.gcr.io/etcd 3.3.10 2c4adeb21b4f 4 months ago 258MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 15 months ago 742kB 在master上查看： # kubectl get nodes NAME STATUS ROLES AGE VERSION vm1.cluster.com Ready master 36m v1.14.0 vm2.cluster.com NotReady <none> 4m36s v1.14.0 如要剔除node节点： # kubectl delete node vm2.cluster.com # kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-fb8b8dccf-4sr5b 1/1 Running 0 36m coredns-fb8b8dccf-rmj7h 1/1 Running 0 36m etcd-vm1.cluster.com 1/1 Running 0 35m kube-apiserver-vm1.cluster.com 1/1 Running 0 35m kube-controller-manager-vm1.cluster.com 1/1 Running 0 35m kube-flannel-ds-amd64-rnght 1/1 Running 0 24m kube-flannel-ds-amd64-sng8b 0/1 Init:0/1 0 4m42s kube-proxy-hptk5 0/1 ContainerCreating 0 4m42s kube-proxy-mxjwr 1/1 Running 0 36m kube-scheduler-vm1.cluster.com 1/1 Running 0 35m # kubectl get pods -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-fb8b8dccf-4sr5b 1/1 Running 0 36m 10.244.0.3 vm1.cluster.com <none> <none> coredns-fb8b8dccf-rmj7h 1/1 Running 0 36m 10.244.0.2 vm1.cluster.com <none> <none> etcd-vm1.cluster.com 1/1 Running 0 35m 192.168.31.11 vm1.cluster.com <none> <none> kube-apiserver-vm1.cluster.com 1/1 Running 0 35m 192.168.31.11 vm1.cluster.com <none> <none> kube-controller-manager-vm1.cluster.com 1/1 Running 0 35m 192.168.31.11 vm1.cluster.com <none> <none> kube-flannel-ds-amd64-rnght 1/1 Running 0 25m 192.168.31.11 vm1.cluster.com <none> <none> kube-flannel-ds-amd64-sng8b 0/1 Init:0/1 0 4m55s 192.168.31.22 vm2.cluster.com <none> <none> kube-proxy-hptk5 0/1 ContainerCreating 0 4m55s 192.168.31.22 vm2.cluster.com <none> <none> kube-proxy-mxjwr 1/1 Running 0 36m 192.168.31.11 vm1.cluster.com <none> <none> kube-scheduler-vm1.cluster.com 1/1 Running 0 36m 192.168.31.11 vm1.cluster.com <none> <none> pod,service,replicaset,deployment,statefulet,daemonset,job,cronjob,node deployment,job：pod的控制器 # kubectl version # kubectl cluster-info Kubernetes master is running at https://192.168.31.11:6443 KubeDNS is running at https://192.168.31.11:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy # kubectl run nginx-deploy --image=nginx:1.14-alpine --port=8080 --replicas=1 --generator=run-pod/v1 kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. deployment.apps/nginx-deploy created (dry run) # kubectl get deployment NAME READY UP-TO-DATE AVAILABLE AGE nginx-deploy 0/1 1 0 69s # kubectl expose (-f FILENAME | TYPE NAME) [--port=port] [--protocol=TCP|UDP|SCTP] [--target-port=number-or-name] [--name=name] [--external-ip=external-ip-of-service] [--type=type] [options] # kubectl expose deployment nginx-deploy --name=nginx --port=80 --target-port=80 --protocol=TCP # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h50m nginx ClusterIP 10.108.178.106 <none> 80/TCP 9s # curl 10.108.178.106 # kubectl describe service nginx # kubectl edit svc nginx #编辑这个service # kubectl scaled --replicas=3 deployment nginx #扩容到3个pod # kubectl describe pods nginx # kubectl set image deployment nginx nginx=nginx:1.15-alpine #更新镜像版本 # kubectl rollout status deployment nginx #查看更新过程，灰度 # kubectl rollout undo deployment nginx #回滚，默认是回滚到上一个版本 在外部访问，需要修改pod的类型 # kubectl edit svc nginx spec: clusterIP: 10.108.178.106 ports: - port: 80 protocol: TCP targetPort: 80 selector: run: nginx-deploy sessionAffinity: None type: ClusterIP --->修改问NodePort # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6h39m nginx NodePort 10.108.178.106 <none> 80:30020/TCP 48m 在浏览器访问：10.108.178.106:30020 # kubectl run myapp --image=ikubernetes/myapp:v1 --replicas=2 # kubectl expose pod myapp --name=myapp --port=80 #实时监视watch # kubectl get pod -w #增加/缩减副本数量： #kubectl scale --replicas=2 deployment myapp #kubectl get pod #升级 #kubectl set image deployment myapp myapp=ikubernetes/v2 #kubectl rollout status deployment myapp #kubectl describe pod myapp-xxxx #回滚 #kubectl rollout undo deployment myapp #kubectl describe pod myapp-xxxx #查看生成的iptabes规则 #iptabes -vnL