Kubernetes服务发布之Ingress

一、什么是Ingress

Ingress是通过service来关联pod的，通过ingress controller实现pod的负载均衡，从而实现全局的负载均衡。

二、Ingress的安装

首先安装helm管理工具：https://helm.sh/docs/intro/install/

使用helm安装ingress：https://kubernetes.github.io/ingress-nginx/deploy/#using-helm

2.1 添加helm仓库

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

2.2 查看并下载helm软件包 

# helm search repo ingress-nginx
NAME                           CHART VERSION    APP VERSION    DESCRIPTION                                       
ingress-nginx/ingress-nginx    4.0.8            1.0.5          Ingress controller for Kubernetes using NGINX a...

# helm pull ingress-nginx/ingress-nginx
# helm pull ingress-nginx/ingress-nginx --version 3.6.0     # 也可以指定版本下载 

2.3 修改对应的配置

# tar xf ingress-nginx-4.0.8.tgz 
# cd ingress-nginx/
# vim values.yaml

a、ingress-nginx/controller、ingress-nginx/kube-webhook-certgen和defaultbackend-amd64镜像地址，需要将k8s.gcr.io仓库中的镜像同步至公司内网镜像仓库
b、将digest注释
c、hostNetwork设置为 true
d、dnsPolicy设置为 ClusterFirstWithHostNet
Default：继承Pod所在宿主机的DNS设置 ClusterFirst：优先使用Kubernetes环境的DNS服务（如CoreDNS提供的域名解析服务），将无法解析的域名转发到从宿主机
ClusterFirstWithHostNet：与ClusterFirst相同，对于以hostNetwork模式运行的Pod，应明确指定使用该策略。
继承的DNS服务器。
None：忽略Kubernetes环境的DNS配置，通过spec.dnsConfig自定义DNS配置。
e、NodeSelector添加ingress: "true"部署至指定节点
f、类型更改为kind: DaemonSet

# sed -n '186p;12p;13p;14p;18p;19p;598p;599p;600p;604p;605p;721p;722p;723p;727p;59p;83p;284p' values.yaml 
  image:
    registry: registry.cn-hangzhou.aliyuncs.com/lzlx
    image: ingress-nginx-controller
    tag: "v1.0.5"
   # digest: sha256:55a1fcda5b7657c372515fe402c3e39ad93aa59f6e4378e82acd99912fe6028d
  dnsPolicy: ClusterFirstWithHostNet
  hostNetwork: true  # 需要修改dnspolicy为
  kind: DaemonSet
  nodeSelector:
      image:
        registry: registry.cn-hangzhou.aliyuncs.com/lzlx
        image: ingress-nginx-kube-webhook-certgen
        tag: v1.1.1
       # digest: sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660
  image:
    registry: estry.cn-hangzhou.aliyuncs.com/lzlx
    image: defaultbackend-amd64
    tag: "1.5"

2.4 部署ingress-nginx，将节点打上label ingress=true，然后将节点IP地址加入到clb负载均衡中

kubectl label node k8s-node3 ingress=true
kubectl create ns ingress-nginx
helm install ingress-nginx -n ingress-nginx .

三、创建一个ingress实例

3.1 创建 nginx web服务

# cat nginx.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: mynginx
        image: nginx:1.18.0
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 500m
            memory: 1024Mi
          limits:
            cpu: 1000m
            memory: 2048Mi
# kubectl create -f nginx.yaml

3.2 创建service

# cat nginx-service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: default
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
  - name: http
    port: 80
    targetPort: 80
# kubectl create -f nginx-service.yaml

3.3 创建ingress

官方参考链接：https://kubernetes.io/zh/docs/concepts/services-networking/ingress/

# cat nginx-ingress.yaml 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress.lzlx.com
spec:
  rules:
  - host: ingress.lzlx.com
    http:
      paths:
      - path: /
        pathType: Prefix         # Exact：精确匹配URL路径，且区分大小写。 Prefix：基于以/ 分隔的 URL 路径前缀匹配。匹配区分大小写，并且对路径中的元素逐个完成。
        backend:
          service:
            name: nginx-service
            port:
              number: 80
  ingressClassName: nginx        # 如果集群版本 >= 1.19使用新ingressClassName: nginx属性代替注释。
# kubectl create -f nginx-ingress.yaml 

3.4 绑定hosts进行访问，ip地址为k8s-node3部署ingress服务的那台机器。

四、ingress的配置

4.1 Redirect，访问域名重定向到baidu.com

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress.lzlx.com
  annotations:
    nginx.ingress.kubernetes.io/permanent-redirect: https://www.baidu.com
spec:
  rules:
  - host: ingress.lzlx.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80
  ingressClassName: nginx

它会自动在ingress-nginx-controller中增加相关的配置

# kubectl exec -it ingress-nginx-controller-5s4fw -n ingress-nginx -- bash
$ cat /etc/nginx/nginx.conf | grep baidu.com
            return 301 https://www.baidu.com;

4.2 rewrite ，访问rewrite.lzlx.com/foo/bar/下的任意内容会重定向到/目录下

# cat nginx-rewrite.yaml 
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rewrite.lzlx.com
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1   # 这里指重定向到/，然后加上所有的参数
spec:
  ingressClassName: nginx
  rules:
  - host: rewrite.lzlx.com
    http:
      paths:
      - path: /foo/bar/(.+)
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80
# kubectl exec -it nginx-deployment-76447d65c-694gr -- bash
root@nginx-deployment-76447d65c-694gr:/# cat /usr/share/nginx/html/b.txt 
this is html dir

 4.3 ssl 证书配置

https://kubernetes.github.io/ingress-nginx/user-guide/tls/

生成证书，生产环境都是购买的通用证书，使用Default SSL Certificate，可以在ingress前面加个nginx proxy将证书配置到nginx proxy上，然后再反向代理到后端的ingress 80端口即可。

上传域名证书，将域名证书创建成secret

# kubectl create secret tls m-test.com --key m-test.com.key --cert m-test.com.crt -n default
secret/m-test.com created

# kubectl get secret
NAME                  TYPE                                  DATA   AGE
default-token-zxp2q   kubernetes.io/service-account-token   3      34d
m-test.com            kubernetes.io/tls                     2      6s

创建ingress文件

# cat nginx-lzlx.m-test.com.yaml 
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: lzlx.m-test.com
spec:
  ingressClassName: nginx
  rules:
  - host: lzlx.m-test.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80
  tls:
  - hosts:
    - lzlx.m-test.com
    secretName: m-test.com

# kubectl create -f nginx-lzlx.m-test.com.yaml 

创建完成再访问的时候会自动https跳转，如果想取消自动跳转加上下面的参数

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: lzlx.medbanks-test.com
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"

4.4 黑白名单配置

Annotations：只对指定的ingress生效

ConfigMap：全局生效

黑名单可以使用ConfigMap去配置，白名单建议使用Annotations去配置。

# cat nginx-lzlx.m-test.com.yaml 
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: lzlx.medbanks-test.com
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "192.168.0.1/24"  # 允许的白名单IP

黑名单配置

# kubectl edit  cm -n ingress-nginx ingress-nginx-controller -oyaml

apiVersion: v1
data:
  allow-snippet-annotations: "true"
  block-cidrs: 120.22.1.33                   # 要加入黑名单的IP地址

# kubectl delete pod -n ingress-nginx --all  # 重启ingress pod，生产慎用

4.5 匹配请求头，如果是移动设备则跳转到m.baidu.com。server-snippet用来写比较复杂的语法

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
        set $agentflag 0;

        if ($http_user_agent ~* "(Mobile)" ){
          set $agentflag 1;
        }

        if ( $agentflag = 1 ) {
          return 301 https://m.baidu.com;
        }

4.6 金丝雀 canary实现灰度发布

创建一个v2版本的ingress，使它的流量和正常的比为50%。

# cat www.test.com-canary.yaml 
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: www.test.com-v2                                    # 和正常的ingress入口做一个区分
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"             # 开启灰度发布，必须先启用canary
    nginx.ingress.kubernetes.io/canary-weight: "50"        # 权重比例，50%的走canary 这个版本
spec:
  ingressClassName: nginx
  rules:
  - host: www.test.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service-v2
            port:
              number: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service-v2
spec:
  type: ClusterIP
  selector:
    app: nginx-v2
  ports:
  - name: http
    port: 80
    targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-v2
  template:
    metadata:
      labels:
        app: nginx-v2
    spec:
      containers:
      - name: mynginx
        image: nginx:1.18.0
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 500m
            memory: 1024Mi
          limits:
            cpu: 1000m
            memory: 2048Mi
# kubectl create -f www.test.com-canary.yaml 
# kubectl exec -it nginx-deployment-v2-6888df4554-pl7vz -- bash
# echo "v2" > /usr/share/nginx/html/index.html        # 将canary版本的主页设置为v2，普通版本的设置为v1
# kubectl get ingress
NAME                     CLASS   HOSTS                    ADDRESS         PORTS     AGE
www.test.com-v1          nginx   www.test.com             10.111.38.227   80        2d23h
www.test.com-v2          nginx   www.test.com             10.111.38.227   80        2d22h

# curl www.test.com  # 进行测试
v2
# curl www.test.com
v2
# curl www.test.com
v1
# curl www.test.com
v1

nginx.ingress.kubernetes.io/canary-by-header：用于通知 Ingress 将请求路由到 Canary Ingress 中指定的服务的标头。当请求头设置为 时always，它将被路由到金丝雀。当标头设置为 时never，它永远不会被路由到金丝雀。对于任何其他值，标头将被忽略，并按优先级将请求与其他金丝雀规则进行比较。

nginx.ingress.kubernetes.io/canary-by-header-value：要匹配的标头值，用于通知 Ingress 将请求路由到 Canary Ingress 中指定的服务。当请求头设置为这个值时，它将被路由到金丝雀。对于任何其他标头值，标头将被忽略，并按优先级将请求与其他金丝雀规则进行比较。此注释必须与 一起使用。注释是 的扩展，nginx.ingress.kubernetes.io/canary-by-header允许自定义标头值而不是使用硬编码值。如果nginx.ingress.kubernetes.io/canary-by-header未定义注释，则没有任何影响。

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: www.test.com-v2
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "user"
    nginx.ingress.kubernetes.io/canary-by-header-value: "leon"

#  curl -H "user: leon" www.test.com
v2
# curl www.test.com
v1

4.7 http登录认证

官方样例：https://kubernetes.github.io/ingress-nginx/examples/auth/basic/

# yum install httpd -y      # 安装htpasswd命令

# htpasswd -c auth leon     # -c 参数创建auth密码文件 (必须是auth否则会报 ingress auth 503 Service Temporarily Unavailable) 用户名是leon输入密码
New password: 
Re-type new password: 
Adding password for user leon

# kubectl create secret generic auth-wwwtest --from-file=auth     # 使用刚生成的auth-wwwtest文件，创建auth-wwwtest 的secret

# kubectl get secret auth-wwwtest -o yaml
apiVersion: v1
data:
  auth-wwwtest: bHpseDokYXByMSR1N2hYZlliZiR2OEFjYmFQcWdRdFYvUlJlNmpMQ3kxCg==
kind: Secret
metadata:
  - apiVersion: v1
  name: auth-wwwtest
  namespace: default
type: Opaque

# cat www.test.com.yaml 
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: www.test.com-v1
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: auth-wwwtest
    nginx.ingress.kubernetes.io/auth-realm: 'describe auth'

# kubectl apply -f www.test.com.yaml