kukubeadm 1.6.1 + docker1.2.6 安装问题

 kubeadm init --apiserver-advertise-address=192.168.20.229 --pod-network-cidr=10.244.0.0/16

kubelet: error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

docker相比1.10增加了KernelMemory变量和CgroupDriver变量，KernelMemory变量表示是否设置linux内核内存限制，CgroupDriver变量表示使用哪个Cgroup驱动，有两种驱动，分别是cgroupfs和systemd，默认使用cgroupfs

由 systemd 变更成 cgroupfs

############################################

或者  --cgroup-driver=systemd 
kubelet的服务配置文件加上这么一行

使用kubeadm 安装 kubernetes1.6.1

 环境准备

master 192.168.20.229
node    192.168.20.223

软件版本：

docker使用 1.12.6

查看版本

yum list kubeadm  --showduplicates |sort -r
kubeadm.x86_64                        1.6.1-0                        kubernetes
kubeadm.x86_64                        1.6.0-0                        kubernetes

yum list kubelet  --showduplicates |sort -r
kubelet.x86_64                        1.6.1-0                        kubernetes
kubelet.x86_64                        1.6.0-0                         kubernetes
kubelet.x86_64                        1.5.4-0                         kubernetes

yum list kubectl  --showduplicates |sort -r
kubectl.x86_64                        1.6.1-0                        kubernetes
kubectl.x86_64                        1.6.0-0                         kubernetes
kubectl.x86_64                        1.5.4-0                         kubernetes

yum list kubernets-cni  --showduplicates |sort -r
kubernetes-cni              x86_64              0.5.1-0                    kubernetes

系统配置

 根据官方文档Installing Kubernetes on Linux with kubeadm中的Limitations小节中的内容，对各节点系统做如下设置:

创建/etc/sysctl.d/k8s.conf文件，添加如下内容：

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

初始化集群

kubeadm init --kubernetes-version=v1.6.1 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.20.229

kubeadm init执行成功后输出下面的信息：

kubeadm init --kubernetes-version=v1.6.1 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.61.41
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.6.1
[init] Using Authorization mode: RBAC
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [node0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.61.41]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 14.583864 seconds
[apiclient] Waiting for at least one node to register
[apiclient] First node has registered after 6.008990 seconds
[token] Using token: e7986d.e440de5882342711
[apiconfig] Created RBAC rules
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run (as a regular user):
  sudo cp /etc/kubernetes/admin.conf $HOME/
  sudo chown $(id -u):$(id -g) $HOME/admin.conf
  export KUBECONFIG=$HOME/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  http://kubernetes.io/docs/admin/addons/
You can now join any number of machines by running the following on each node
as root:

kubeadm join --token 881f96.aaf02f1f8dc53889 192.168.20.229:6443

Master Node初始化完成，使用kubeadm初始化的Kubernetes集群在Master节点上的核心组件：kube-apiserver,kube-scheduler, kube-controller-manager是以静态Pod的形式运行的。

ls /etc/kubernetes/manifests/
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

在/etc/kubernetes/manifests/目录里可以看到kube-apiserver,kube-scheduler, kube-controller-manager的定义文件。另外集群持久化存储etcd也是以单点静态Pod的形式运行的，对于etcd后边我们会把它切换成etcd集群。

查看一下kube-apiserver.yaml的内容：

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    .......
    - --insecure-port=0

注意到kube-apiserver的选项--insecure-port=0，也就是说kubeadm 1.6.0初始化的集群，kube-apiserver没有监听默认的http 8080端口。

所以我们使用kubectl get nodes会报The connection to the server localhost:8080 was refused - did you specify the right host or port?。

查看kube-apiserver的监听端口可以看到只监听了https的6443端口

netstat -nltp | grep apiserver
tcp6       0      0 :::6443                 :::*                    LISTEN      9831/kube-apiserver

为了使用kubectl访问apiserver，在~/.bash_profile中追加下面的环境变量：

export KUBECONFIG=/etc/kubernetes/admin.conf

source ~/.bash_profile

此时kubectl命令在master node上就好用了，查看一下当前机器中的Node：

kubectl get nodes
NAME      STATUS     AGE       VERSION
k8s1       NotReady   3m        v1.6.1

安装Pod Network

接下来安装flannel network add-on：

kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
kubectl apply -f  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
serviceaccount "flannel" created
configmap "kube-flannel-cfg" created
daemonset "kube-flannel-ds" created

如果Node有多个网卡的话，参考flannel issues 39701，目前需要在kube-flannel.yml中使用--iface参数指定集群主机内网网卡的名称，否则可能会出现dns无法解析。需要将kube-flannel.yml下载到本地，flanneld启动参数加上--iface=<iface-name>

......
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
......
containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.7.0-amd64
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=eth1" ]
......

使用kubectl get pod --all-namespaces -o wide确保所有的Pod都处于Running状态

kubectl get pod --all-namespaces -o wide

或者

kubectl --kubeconfig=/etc/kubernetes/admin.conf get pod  --all-namespaces -o wide

NAMESPACE     NAME                           READY     STATUS    RESTARTS   AGE       IP               NODE
kube-system   etcd-k8s1                      1/1       Running   0          10m       192.168.20.229   k8s1
kube-system   kube-apiserver-k8s1            1/1       Running   0          10m       192.168.20.229   k8s1
kube-system   kube-controller-manager-k8s1   1/1       Running   0          10m       192.168.20.229   k8s1
kube-system   kube-dns-3913472980-g97bm      3/3       Running   0          10m       10.244.1.2       k8s5
kube-system   kube-flannel-ds-k87tt          2/2       Running   0          2m        192.168.20.233   k8s5
kube-system   kube-flannel-ds-lq62q          2/2       Running   0          2m        192.168.20.229   k8s1
kube-system   kube-proxy-0nrp0               1/1       Running   0          10m       192.168.20.229   k8s1
kube-system   kube-proxy-qcds5               1/1       Running   0          10m       192.168.20.233   k8s5
kube-system   kube-scheduler-k8s1            1/1       Running   0          10m       192.168.20.229   k8s1

使master node参与工作负载

使用kubeadm初始化的集群，出于安全考虑Pod不会被调度到Master Node上，也就是说Master Node不参与工作负载。

这里搭建的是测试环境可以使用下面的命令使Master Node参与工作负载：

kubectl taint nodes --all  node-role.kubernetes.io/master-

测试DNS

[root@k8s1 ~]# kubectl --kubeconfig=/etc/kubernetes/admin.conf run curl --image=radial/busyboxplus:curl -i --tty
If you don't see a command prompt, try pressing enter.
[ root@curl-57077659-s2l5v:/ ]$ nslookup
BusyBox v1.22.1 (2014-09-13 22:15:30 PDT) multi-call binary.

Usage: nslookup [HOST] [SERVER]

Query the nameserver for the IP address of the given HOST
optionally using a specified DNS server

[ root@curl-57077659-s2l5v:/ ]$ nslookup kube-dns.kube-system
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kube-dns.kube-system
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
[ root@curl-57077659-s2l5v:/ ]$ nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

测试OK后，删除掉curl这个Pod。

kubectl delete deploy curl

向集群中添加节点

kubeadm join --token 881f96.aaf02f1f8dc53889 192.168.20.229:6443

查看集群中节点：

[root@k8s1 ~]# kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes
NAME      STATUS    AGE       VERSION
k8s1      Ready     54m       v1.6.1
k8s5      Ready     54m       v1.6.1

安装Dashboard插件

wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml

kubectl create -f kubernetes-dashboard.yaml

从http://NodeIp:NodePort访问dashboard，浏览器显示下面的错误

User "system:serviceaccount:kube-system:default" cannot list statefulsets.apps in the namespace "default". (get statefulsets.apps)



这是因为Kubernetes 1.6开始API Server启用了RBAC授权，当前的kubernetes-dashboard.yaml没有定义授权的ServiceAccount，所以访问API Server时被拒绝了。

根据https://github.com/kubernetes/dashboard/issues/1803中的内容临时授予system:serviceaccount:kube-system:default cluster_admin的角色，临时解决一下。

创建dashboard-rbac.yaml，定义system:serviceaccount:kube-system:default和ClusterRole cluster-admin绑定：

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: dashboard-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin 
subjects:
- kind: ServiceAccount
  name: default
  namespace: kube-system

kubectl  --kubeconfig=/etc/kubernetes/admin.conf create -f dashboard-rbac.yml

在集群中运行Heapster

下面安装Heapster为集群添加使用统计和监控功能，为Dashboard添加仪表盘。

下载最新的Heapster到集群中的某个Node上

wget https://github.com/kubernetes/heapster/archive/v1.3.0.tar.gz

使用InfluxDB做为Heapster的后端存储，开始部署，中间会pull相关镜像，包含gcr.io/google_containers/heapster_grafana:v2.6.0-2

tar -zxvf v1.3.0.tar.gz
cd heapster-1.3.0/deploy/kube-config/influxdb 

添加了RBAC授权

[root@k8s1 influxdb]# cat heapster-rbac.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: heapster
  namespace: kube-system

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: heapster
subjects:
  - kind: ServiceAccount
    name: heapster
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:heapster
  apiGroup: rbac.authorization.k8s.io

[root@k8s1 influxdb]# vim heapster-deployment.yaml 

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        image: gcr.io/google_containers/heapster-amd64:v1.3.0-beta.1
        imagePullPolicy: IfNotPresent
        command:
        - /heapster
        - --source=kubernetes:https://kubernetes.default
        - --sink=influxdb:http://monitoring-influxdb:8086

参考

http://blog.frognew.com/2017/04/kubeadm-install-kubernetes-1.6.html

https://github.com/opsnull/follow-me-install-kubernetes-cluster/blob/master/10-%E9%83%A8%E7%BD%B2Heapster%E6%8F%92%E4%BB%B6.md