HPA

目录

• HPA
  • 创建HPA
    • 创建Metrics-Server
    • 创建HPA

HPA

HPA（Horizontal Pod Autoscaler，水平Pod自动伸缩器）可根据观察到的CPU、内存使用率或自定义度量标准来自动扩展或缩容Pod的数量。HPA不适用于无法缩放的对象，比如DaemonSet。

HPA控制器会定期调整RC或Deployment的副本数，以使观察到的平均CPU利用率与用户指定的目标相匹配。

HPA需要metrics-server（项目地址：https://github.com/kubernetes-incubator/metrics-server ）获取度量指标，由于在高可用集群安装中已经安装了metrics-server，所以本节的实践部分无须再次安装。

创建HPA

在生产环境中，总会有一些意想不到的事情发生，比如公司网站流量突然升高，此时之前创建的Pod已不足以撑住所有的访问，而运维人员也不可能24小时守着业务服务，这时就可以通过配置HPA，实现负载过高的情况下自动扩容Pod副本数以分摊高并发的流量，当流量恢复正常后，HPA会自动缩减Pod的数量。

创建Metrics-Server

如果要使用HAP就必须使用metrics-server来获取主机信息。

• 创建证书

[root@kubernetes-master-01 k8s]# cat > front-proxy-ca-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
     "algo": "rsa",
     "size": 2048
  }
}
EOF
[root@kubernetes-master-01 k8s]# cat > front-proxy-client-csr.json <<EOF
{
  "CN": "front-proxy-client",
  "key": {
     "algo": "rsa",
     "size": 2048
  }
}
EOF
[root@kubernetes-master-01 k8s]# cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare front-proxy-ca
2020/09/29 17:34:10 [INFO] generating a new CA key and certificate from CSR
2020/09/29 17:34:10 [INFO] generate received request
2020/09/29 17:34:10 [INFO] received CSR
2020/09/29 17:34:10 [INFO] generating key: rsa-2048
2020/09/29 17:34:10 [INFO] encoded CSR
2020/09/29 17:34:10 [INFO] signed certificate with serial number 201379696820699389255209103483926865101225990414
[root@kubernetes-master-01 k8s]# cfssl gencert   
> -ca=./front-proxy-ca.pem   
> -ca-key=./front-proxy-ca-key.pem   
> -config=./ca-config.json   
> -profile=kubernetes   
> front-proxy-client-csr.json | cfssljson -bare front-proxy-client
2020/09/29 17:35:22 [INFO] generate received request
2020/09/29 17:35:22 [INFO] received CSR
2020/09/29 17:35:22 [INFO] generating key: rsa-2048
2020/09/29 17:35:22 [INFO] encoded CSR
2020/09/29 17:35:22 [INFO] signed certificate with serial number 182837546809643737009256039519642110908880768521
2020/09/29 17:35:22 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@kubernetes-master-01 k8s]# cp front-proxy-c*pem /etc/kubernetes/pki/
[root@kubernetes-master-01 k8s]# ll /etc/kubernetes/pki/
total 28
-rw------- 1 root root 1679 Sep 28 21:38 ca-key.pem
-rw-r--r-- 1 root root  960 Sep 28 21:38 ca.csr
-rw-r--r-- 1 root root 1273 Sep 28 21:38 ca.pem
-rw------- 1 root root 1675 Sep 29 18:04 front-proxy-ca-key.pem
-rw-r--r-- 1 root root 1143 Sep 29 18:04 front-proxy-ca.pem
-rw------- 1 root root 1675 Sep 29 18:04 front-proxy-client-key.pem
-rw-r--r-- 1 root root 1188 Sep 29 18:04 front-proxy-client.pem
[root@kubernetes-master-01 ~]# kubectl get pods -n kube-system
NAME                              READY   STATUS    RESTARTS   AGE
coredns-7b5c749bf5-l5ks2          1/1     Running   0          27h
metrics-server-78c5d9dfff-2v8fd   1/1     Running   0          12sCopy to clipboardErrorCopied

• 安装metrics-server

[root@kubernetes-master-01 ~]# cat > components.yaml <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:aggregated-metrics-reader
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      - name: ca-ssl
        hostPath:
         path: /etc/kubernetes/pki
      containers:
      - name: metrics-server
        image: registry.cn-hangzhou.aliyuncs.com/k8sos/metrics-server:v0.3.7
        imagePullPolicy: IfNotPresent
        args:
          - --cert-dir=/tmp
          - --secure-port=4443
          - --metric-resolution=30s
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
          - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem
          - --requestheader-username-headers=X-Remote-User
          - --requestheader-group-headers=X-Remote-Group
          - --requestheader-extra-headers-prefix=X-Remote-Extra-
        ports:
        - name: main-port
          containerPort: 4443
          protocol: TCP
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
        - name: ca-ssl
          mountPath: /etc/kubernetes/pki
      nodeSelector:
        kubernetes.io/os: linux
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: main-port
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
EOF
[root@kubernetes-master-01 ~]# kubectl delete -f components.yaml
clusterrole.rbac.authorization.k8s.io "system:aggregated-metrics-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "metrics-server:system:auth-delegator" deleted
rolebinding.rbac.authorization.k8s.io "metrics-server-auth-reader" deleted
apiservice.apiregistration.k8s.io "v1beta1.metrics.k8s.io" deleted
serviceaccount "metrics-server" deleted
deployment.apps "metrics-server" deleted
service "metrics-server" deleted
clusterrole.rbac.authorization.k8s.io "system:metrics-server" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:metrics-server" deletedCopy to clipboardErrorCopied

• 在kube-apiserver配置中加入metrics-server证书

# /etc/kubernetes/cfg/kube-apiserver.conf
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  Copy to clipboardErrorCopied

• 验证

[root@kubernetes-master-01 ~]# kubectl top pods -n kube-system
NAME                              CPU(cores)   MEMORY(bytes)
coredns-7b5c749bf5-l5ks2          3m           15Mi
metrics-server-589847c86f-cg6d4   1m           11MiCopy to clipboardErrorCopied

创建HPA

HPA是根据CPU的使用率、内存使用率自动扩展Pod数量的，所以要使用HPA就必须定义Requests参数

kind: Deployment
apiVersion: apps/v1
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
          resources:
            limits:
              cpu: 10m
              memory: 50Mi
            requests:
              cpu: 10m
              memory: 50Mi
---
kind: Service
apiVersion: v1
metadata:
  name: svc
  namespace: default
spec:
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80
---
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: docs
  namespace: default
spec:
  maxReplicas: 10
  minReplicas: 1
  scaleTargetRef:
    kind: Deployment
    name: nginx
    apiVersion: apps/v1
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 5Copy to clipboardErrorCopied

• 查看

[root@kubernetes-master-01 ~]# kubectl get -f hpa.yaml
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1           20m

NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/svc   ClusterIP   10.96.119.88   <none>        80/TCP    20m

NAME                                       REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/docs   Deployment/nginx   0%/5%     1         10        1          20mCopy to clipboardErrorCopied

• 加压

[root@kubernetes-master-01 ~]# ab -c 1000 -n 1000 http://10.96.119.88/index.html
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.96.119.88 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx/1.19.2
Server Hostname:        10.96.119.88
Server Port:            80

Document Path:          /index.html
Document Length:        612 bytes

Concurrency Level:      1000
Time taken for tests:   6.389 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      845000 bytes
HTML transferred:       612000 bytes
Requests per second:    156.51 [#/sec] (mean)
Time per request:       6389.404 [ms] (mean)
Time per request:       6.389 [ms] (mean, across all concurrent requests)
Transfer rate:          129.15 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0  230 749.0      1    3051
Processing:    30  879 738.9    700    3956
Waiting:        0  875 739.0    700    3955
Total:         34 1108 1351.2    701    6378

Percentage of the requests served within a certain time (ms)
  50%    701
  66%    799
  75%    800
  80%    895
  90%   2800
  95%   5150
  98%   5349
  99%   6378
 100%   6378 (longest request)Copy to clipboardErrorCopied

• 查看结果

[root@kubernetes-master-01 ~]# kubectl get pods -w -o wide
NAME                     READY   STATUS              RESTARTS   AGE   IP             NODE                   NOMINATED NODE   READINESS GATES
nginx-6cf7488b57-992qk   0/1     ContainerCreating   0          43s   <none>         kubernetes-master-01   <none>           <none>
nginx-6cf7488b57-bd887   1/1     Running             0          21m   10.243.24.2    kubernetes-master-01   <none>           <none>
nginx-6cf7488b57-h8gk6   1/1     Running             0          58s   10.243.24.14   kubernetes-master-01   <none>           <none>
nginx-6cf7488b57-hfr7b   1/1     Running             0          58s   10.243.24.12   kubernetes-master-01   <none>           <none>
nginx-6cf7488b57-sbtz9   1/1     Running             0          59s   10.243.24.13   kubernetes-master-01   <none>           <none>
nginx-6cf7488b57-xg8sb   0/1     ContainerCreating   0          43s   <none>         kubernetes-master-01   <none>           <none>
nginx-6cf7488b57-992qk   1/1     Running             0          43s   10.243.24.16   kubernetes-master-01   <none>           <none>
[root@kubernetes-master-01 ~]# kubectl get hpa -w
NAME   REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
docs   Deployment/nginx   0%/5%     1         10        6          22m
docs   Deployment/nginx   5%/5%     1         10        6          22m
docs   Deployment/nginx   11%/5%    1         10        6          23m
docs   Deployment/nginx   45%/5%    1         10        10         23mCopy to clipboardErrorCopied