kubernetes 安装 helm，ingress

1、安装helm客户端

[root@cc-k8s01 work]# wget https://storage.googleapis.com/kubernetes-helm/helm-v2.14.0-rc.2-linux-amd64.tar.gz
[root@cc-k8s01 work]# tar zxf helm-v2.14.0-rc.2-linux-amd64.tar.gz
[root@cc-k8s01 work]# cd linux-amd64/
[root@cc-k8s01 linux-amd64]# mv helm /opt/k8s/bin
[root@cc-k8s01 linux-amd64]# helm
The Kubernetes package manager
 
To begin working with Helm, run the 'helm init' command:
 
    $ helm init

2、安装tiller服务端

1.创建RBACq角色并授权

kubectl create serviceaccount tiller --namespace kube-system
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

　　2.创建tls通迅的证书

##创建tiller服务器证书CSR
cd /opt/k8s/work
cat > tiller-server-csr.json <<EOF
{
  "CN": "tiller-server",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
     {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF
##生成证书，沿用kubernetes的CA证书
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem 
  -ca-key=/opt/k8s/work/ca-key.pem 
  -config=/opt/k8s/work/ca-config.json 
  -profile=kubernetes tiller-server-csr.json | cfssljson -bare tiller-server
ls tiller-server*

####创建helm与tiller服务器通迅的证书CSR
cd /opt/k8s/work
cat > helm-csr.json <<EOF
{
  "CN": "tiller",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
     {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF
##生成证书，沿用kubernetes的CA证书
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem 
  -ca-key=/opt/k8s/work/ca-key.pem 
  -config=/opt/k8s/work/ca-config.json 
  -profile=kubernetes helm-csr.json | cfssljson -bare helm

　　3.初始化tiller

#由于镜像是国外镜像，故需先使用其他源
[root@cc-k8s02 ~]# docker pull gcr.azk8s.cn/kubernetes-helm/tiller:v2.14.0-rc.2
v2.14.0-rc.2: Pulling from kubernetes-helm/tiller
bdf0201b3a05: Pull complete 
879b5272666d: Pull complete 
3dd50e1bb957: Pull complete 
ba6c28ffc2cd: Pull complete 
Digest: sha256:5d19651f555dfcd9aafa52ee569413aa38e60ddab19e54c6dc33ad60da2d46ed
Status: Downloaded newer image for gcr.azk8s.cn/kubernetes-helm/tiller:v2.14.0-rc.2
[root@cc-k8s02 ~]# docker tag gcr.azk8s.cn/kubernetes-helm/tiller:v2.14.0-rc.2 gcr.io/kubernetes-helm/tiller:v2.14.0-rc.2

[root@cc-k8s01 work]# helm init --tiller-tls --tiller-tls-cert ./tiller-server.pem --tiller-tls-key ./tiller-server-key.pem --tiller-tls-verify --tls-ca-cert ca.pem
$HELM_HOME has been configured at /root/.helm.

[root@cc-k8s01 work]# kubectl get pods -n kube-system
NAME                                    READY   STATUS    RESTARTS   AGE
coredns-5b969f4c88-n8kht                1/1     Running   4          9d
elasticsearch-logging-0                 1/1     Running   0          9d
elasticsearch-logging-1                 1/1     Running   0          9d
fluentd-es-v2.4.0-5n9q2                 1/1     Running   0          9d
fluentd-es-v2.4.0-8flsx                 1/1     Running   0          9d
fluentd-es-v2.4.0-qlhb5                 1/1     Running   0          9d
kibana-logging-f4d99b69f-mw28c          1/1     Running   0          9d
kube-state-metrics-699fdf75f8-2tskt     1/1     Running   0          2d
kubernetes-dashboard-7848d45466-mg4w5   1/1     Running   0          9d
metrics-server-6f97f5879-qg8vx          1/1     Running   0          9d
tiller-deploy-78fc7f6db4-jq5zs          1/1     Running   0          14s

##将helm的证书、私钥和CA证书拷贝到/root/.helm，并改名为key.pem和cert.pem
cp ca.pem helm.pem helm-key.pem /root/.helm
cd /root/.helm
mv helm-key.pem key.pem
mv helm.pem cert.pem

##然后就可以使用helm 加上--tls参数与tiller进行加密通迅
[root@cc-k8s01 work]# helm ls --tls

产生以下报错
[root@cc-k8s01 work]# helm ls --tls
E0703 13:32:38.232485   22034 portforward.go:400] an error occurred forwarding 42494 -> 44134: error forwarding port 44134 to pod c656f2021e1cdea7ac0710bf13d9e55d27a7713f4c4c77f80b92379be5d544bf, uid : unable to do port forwarding: socat not found.
E0703 13:32:39.236986   22034 portforward.go:400] an error occurred forwarding 42494 -> 44134: error forwarding port 44134 to pod c656f2021e1cdea7ac0710bf13d9e55d27a7713f4c4c77f80b92379be5d544bf, uid : unable to do port forwarding: socat not found.
E0703 13:32:40.688934   22034 portforward.go:400] an error occurred forwarding 42494 -> 44134: error forwarding port 44134 to pod c656f2021e1cdea7ac0710bf13d9e55d27a7713f4c4c77f80b92379be5d544bf, uid : unable to do port forwarding: socat not found.
E0703 13:33:13.681938   22034 portforward.go:340] error creating error stream for port 42494 -> 44134: Timeout occured

[root@cc-k8s01 work]:~# helm version
Client: &version.Version{SemVer:"v2.14.0-rc.2", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}
E0711 10:09:50.160064   10916 portforward.go:332] an error occurred forwarding 33491 -> 44134: error forwarding port 44134 to pod tiller-deploy-542252878-15h67_kube-system, uid : unable to do port forwarding: socat not found.
Error: cannot connect to Tiller

！！解决yum install -y socat

[root@cc-k8s01 work]# helm ls --tls
Error: context deadline exceeded
[root@cc-k8s01 nginx-ingress]# helm reset
##重新授权
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
[root@cc-k8s01 work]# helm init --tiller-tls --tiller-tls-cert ./tiller-server.pem --tiller-tls-key ./tiller-server-key.pem --tiller-tls-verify --tls-ca-cert ca.pem
$HELM_HOME has been configured at /root/.helm.　

3、安装ingress

1、下载Charts

由于helm的默认charts repository是 https://kubernetes-charts.storage.googleapis.com/ 无法访问，所以需通过离线安装的方式先从github.com clone相关的charts

[root@cc-k8s01 work]# git clone https://github.com/helm/charts.git
Cloning into 'charts'...
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 69700 (delta 4), reused 8 (delta 4), pack-reused 69689
Receiving objects: 100% (69700/69700), 19.88 MiB | 4.27 MiB/s, done.
Resolving deltas: 100% (49888/49888), done.
 
[root@cc-k8s01 work]# cd charts/stable/nginx-ingress/
[root@cc-k8s01 nginx-ingress]# ls -l
total 48
-rw-r--r-- 1 root root   569 Jun  4 17:21 Chart.yaml
drwxr-xr-x 2 root root  4096 Jun  4 17:21 ci
-rw-r--r-- 1 root root   110 Jun  4 17:21 OWNERS
-rw-r--r-- 1 root root 17923 Jun  4 17:21 README.md
drwxr-xr-x 2 root root  4096 Jun  4 17:26 templates
-rw-r--r-- 1 root root 10045 Jun  4 17:21 values.yaml

2、下载必要的镜像

nginx-ingress的镜像是从quay.io拉取，可以正常访问。但是defaultbackend的镜像是从k8s.gcr.io拉取，这个没法访问。需要先从dockerhub上拉取，不然部署的时候会报ErrImagePull

[root@cc-k8s02 ~]# docker pull googlecontainer/defaultbackend-amd64:1.5
1.5: Pulling from googlecontainer/defaultbackend-amd64
65f4220de95d: Pull complete 
Digest: sha256:4dc5e07c8ca4e23bddb3153737d7b8c556e5fb2f29c4558b7cd6e6df99c512c7
Status: Downloaded newer image for googlecontainer/defaultbackend-amd64:1.5
[root@cc-k8s02 ~]# docker tag googlecontainer/defaultbackend-amd64:1.5 k8s.gcr.io/defaultbackend-amd64:1.5

3、通过helm安装nginx-ingress

常用参数如下：

• controller.hostNetwork=true hostNetwork模式，即在node主机上监听80/443端口用来集群外提供服务。默认为nodePort模式
• controller.dnsPolicy DNS策略，如果使用hostNetwork，则必须配置为ClusterFirstWithHostNet。目的是允许nginx-ingress使用主机网络监听，但仍沿用集群DNS解析
• controller.stats.enabled=true 开启nginx的status页面，以获取nginx运行的统计信息
• controller.metrics.enabled=true 开户prometheus监控metrics
• rbac.create=true 部署时创建nginx-ingress所需的rbac角色和权限，在使用rbac权限体系的k8s集群中必须指定此项
• controller.kind 部署为Deployment或者DaemonSet，如果是Deployment则默认只部署一个，如果是DaemonSet则在每个Node上部署一个

安装操作

[root@cc-k8s01 nginx-ingress]# helm install --tls --namespace kube-system --name nginx-ingress --set controller.kind=DaemonSet --set rbac.create=true --set controller.stats.enabled=true --set controller.metri controller.dnsPolicy=ClusterFirstWithHostNet ./
NAME:   nginx-ingress
LAST DEPLOYED: Wed Jul  3 14:25:54 2019
NAMESPACE: kube-system
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                                            READY  STATUS             RESTARTS  AGE
nginx-ingress-controller-2rlpx                  0/1    ContainerCreating  0         0s
nginx-ingress-controller-8lpbs                  0/1    ContainerCreating  0         0s
nginx-ingress-controller-zh4z9                  0/1    ContainerCreating  0         0s
nginx-ingress-default-backend-7474b6b4cd-d6d9d  0/1    ContainerCreating  0         0s

==> v1/Service
NAME                              TYPE          CLUSTER-IP      EXTERNAL-IP  PORT(S)                     AGE
nginx-ingress-controller          LoadBalancer  10.254.51.167   <pending>    80:31220/TCP,443:30851/TCP  0s
nginx-ingress-controller-metrics  ClusterIP     10.254.70.26    <none>       9913/TCP                    0s
nginx-ingress-controller-stats    ClusterIP     10.254.250.132  <none>       18080/TCP                   0s
nginx-ingress-default-backend     ClusterIP     10.254.142.218  <none>       80/TCP                      0s

==> v1/ServiceAccount
NAME           SECRETS  AGE
nginx-ingress  1        0s

==> v1beta1/ClusterRole
NAME           AGE
nginx-ingress  0s

==> v1beta1/ClusterRoleBinding
NAME           AGE
nginx-ingress  0s

==> v1beta1/DaemonSet
NAME                      DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
nginx-ingress-controller  3        3        0      3           0          <none>         0s

==> v1beta1/Deployment
NAME                           READY  UP-TO-DATE  AVAILABLE  AGE
nginx-ingress-default-backend  0/1    1           0          0s

==> v1beta1/Role
NAME           AGE
nginx-ingress  0s

==> v1beta1/RoleBinding
NAME           AGE
nginx-ingress  0s


NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace kube-system get services -o wide -w nginx-ingress-controller'

An example Ingress that makes use of the controller:

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

上面安装使用了hostNetwork，可以看到在每个Node监听了80/443

[root@cc-k8s02 ~]# netstat -nltp|grep -E "(:80|:443)"
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      13676/nginx: master 
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      13676/nginx: master

controller.stats.enabled=true 参数开启了nginx_status页面，可以通过curl来验证nginx-ingress的联通性

[root@cc-k8s02 ~]# curl localhost/nginx_status
Active connections: 1 
server accepts handled requests
 1395 1395 1392 
Reading: 0 Writing: 1 Waiting: 0

4、通过helm删除nginx-ingress部署

如果不带--purge，则名叫nginx-ingress仍会存在，只是状态为delete。下次如果使用同名部署则会报错，使用–purge是完整删除。

[root@cc-k8s01 ~]# helm delete --tls  nginx-ingress --purge
release "nginx-ingress" deleted