Kubernetes 部署 Nginx Ingress Controller 之 nginxinc/kubernetes-ingress

更新：这里用的是 nginxinc/kubernetes-ingress ，还有个 kubernetes/ingress-nginx ，它们的区别见 Differences Between nginxinc/kubernetes-ingress and kubernetes/ingress-nginx Ingress Controllers ，后来我们选用了 kubernetes/ingress-nginx ，详见博文。

开始天真地以为只要写一个 ingress 配置文件并部署好就行了。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cnblogs-ingress
spec:
  rules:
  - host: q.cnblogs.com
    http:
      paths:
        - backend:
            serviceName: q-web
            servicePort: 80

# kubectl apply -f cnblogs-ingress.yaml
# kubectl get ingress
NAME              HOSTS           ADDRESS   PORTS   AGE
cnblogs-ingress   q.cnblogs.com             80      6h18

但部署后发现所有 node 服务器上没有任何进程监听 80 端口，显然不对。

从 k8s 帮助文档中知道了答案：

You must have an ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect.
In order for the Ingress resource to work, the cluster must have an ingress controller running.
Unlike other types of controllers which run as part of the kube-controller-manager binary, Ingress controllers are not started automatically with a cluster. Use this page to choose the ingress controller implementation that best fits your cluster.

原来 k8s 没有内置 ingress controller ，需要安装第三方的 ingress controller ，比如 nginx ingress controller ，上面通过 cnblogs-ingress.yaml 只是创建了 ingress 资源。那为什么通过 deployment.yaml 创建了 deployment 资源就能正常部署 pod ？那是因为 kube-controller-manager 中内置了 deployment controller 。

我们选用 nginx ingress controller ，部署操作步骤如下（参考文档）：
1）从 github 上签出 kubernetes-ingress 仓库

$ git clone https://github.com/nginxinc/kubernetes-ingress/
$ cd kubernetes-ingress
$ git checkout v1.6.1 -f
$ cd deployments

2）创建 namespace 与 ServiceAccount ，都叫 nginx-ingress

kubectl apply -f common/ns-and-sa.yaml

3）创建 cluster role 与 cluster role binding

kubectl apply -f rbac/rbac.yaml

4）创建 secret
使用自己的证书文件创建 secret

kubectl create secret tls default-server-secret --cert=path/to/cert.pem --key=path/to/key.pem

或者使用 nginx-ingress 自带的证书创建 sescret

kubectl apply -f common/default-server-secret.yaml

5）创建 ConfigMap

kubectl apply -f common/nginx-config.yaml 

6）创建 custom resource definitions

kubectl apply -f common/custom-resource-definitions.yaml

7）创建 DaemonSet

kubectl apply -f daemon-set/nginx-ingress.yaml

8）查看 pod 是否部署成功

$ kubectl get pods --namespace=nginx-ingress                                                                                                1 ↵
NAME                  READY   STATUS    RESTARTS   AGE
nginx-ingress-7xdzp   1/1     Running   5          12m
nginx-ingress-rs4th   1/1     Running   0          114s
nginx-ingress-w2fnh   1/1     Running   0          12m
nginx-ingress-z54r6   1/1     Running   5          12m

9）创建监听 31080 端口的 NodePort 类型的 service

配置文件 nodeport.yaml （去掉了443端口）
注0：nodePort 只能使用 30000-32767 范围的端口。
注1：去掉了443端口，我们在最前端使用了阿里云负载均衡，请求都通过 http 转发。

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  type: NodePort 
  ports:
  - nodePort: 31080
     port: 80
     targetPort: 80
     protocol: TCP
     name: http
  selector:
    app: nginx-ingress

部署命令

kubectl apply -f service/nodeport.yaml

10）检查 nginx-ingress 部署成功

进入 nginx-ingress 容器

kubectl exec -it daemonset/nginx-ingress -n nginx-ingress /bin/bash

查看 nginx 配置

cat /etc/nginx/conf.d/production-cnblogs-ingress.conf

确认 ingress 中添加的转发规则已被导入

upstream production-cnblogs-ingress-q.cnblogs.com-q-web-80 {
	zone production-cnblogs-ingress-q.cnblogs.com-q-web-80 256k;
	random two least_conn;
	
	server 192.168.107.211:80 max_fails=1 fail_timeout=10s max_conns=0;
	server 192.168.186.72:80 max_fails=1 fail_timeout=10s max_conns=0;	
}

server {
    listen 80;
	server_tokens on;
	server_name q.cnblogs.com;
	
	location / {
		proxy_http_version 1.1;
		proxy_connect_timeout 60s;
		proxy_read_timeout 60s;
		proxy_send_timeout 60s;
		client_max_body_size 1m;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Host $host;
		proxy_set_header X-Forwarded-Port $server_port;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_buffering on;
		
		proxy_pass http://production-cnblogs-ingress-q.cnblogs.com-q-web-80;
	}
}

至此 nginx-ingress 部署成功。

解决转发 X-Forwarded-Proto 请求头问题

解决方法：在 ingress 配置文件中添加 nginx.org/redirect-to-https: "true"，详见博问 K8s Nginx Ingress Controller 转发 X-Forwarded-Proto 请求头的问题

apiVersion: extensions/v1beta1 
kind: Ingress
metadata:
  name: cnblogs-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.org/redirect-to-https: "true"

添加 proxy_set_header 配置

通过 ingress 的 nginx.org/location-snippets 注解添加如下的配置：

apiVersion: extensions/v1beta1 
kind: Ingress
metadata:
  name: cnblogs-ingress
  annotations:
    nginx.org/location-snippets: |
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;

七层负载均衡对 nginx-ingress 进行健康检查问题

详见博问 阿里云负载均衡对 K8s Nginx Ingress 的健康检查问题

基于二级域名自动转发到 service 的实现方法

详见博问 K8s Ingress 如何自动根据主机名中的二级域名匹配 service

应用获取不到客户端真实 IP 地址的问题

详见博问 K8s 中 ASP.NET Core 应用获取不到客户端真实 IP 地址