HPA
HPA（Horizontal Pod Autoscaler，水平Pod自动伸缩器）可根据观察到的CPU、内存使用率或自定义度量标准来自动扩展或缩容Pod的数量。HPA不适用于无法缩放的对象，比如DaemonSet。
HPA控制器会定期调整RC或Deployment的副本数，以使观察到的平均CPU利用率与用户指定的目标相匹配。
HPA需要metrics-server（项目地址：https://github.com/kubernetes-incubator/metrics-server ）获取度量指标，由于在高可用集群安装中已经安装了metrics-server，所以本节的实践部分无须再次安装。
创建HPA
在生产环境中，总会有一些意想不到的事情发生，比如公司网站流量突然升高，此时之前创建的Pod已不足以撑住所有的访问，而运维人员也不可能24小时守着业务服务，这时就可以通过配置HPA，实现负载过高的情况下自动扩容Pod副本数以分摊高并发的流量，当流量恢复正常后，HPA会自动缩减Pod的数量。
创建Metrics-Server
如果要使用
HAP就必须使用metrics-server来获取主机信息。
- 创建证书
[root@kubernetes-master-01 k8s]# cat > front-proxy-ca-csr.json <<EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
}
}
EOF
[root@kubernetes-master-01 k8s]# cat > front-proxy-client-csr.json <<EOF
{
"CN": "front-proxy-client",
"key": {
"algo": "rsa",
"size": 2048
}
}
EOF
[root@kubernetes-master-01 k8s]# cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare front-proxy-ca
2020/09/29 17:34:10 [INFO] generating a new CA key and certificate from CSR
2020/09/29 17:34:10 [INFO] generate received request
2020/09/29 17:34:10 [INFO] received CSR
2020/09/29 17:34:10 [INFO] generating key: rsa-2048
2020/09/29 17:34:10 [INFO] encoded CSR
2020/09/29 17:34:10 [INFO] signed certificate with serial number 201379696820699389255209103483926865101225990414
[root@kubernetes-master-01 k8s]# cfssl gencert
> -ca=./front-proxy-ca.pem
> -ca-key=./front-proxy-ca-key.pem
> -config=./ca-config.json
> -profile=kubernetes
> front-proxy-client-csr.json | cfssljson -bare front-proxy-client
2020/09/29 17:35:22 [INFO] generate received request
2020/09/29 17:35:22 [INFO] received CSR
2020/09/29 17:35:22 [INFO] generating key: rsa-2048
2020/09/29 17:35:22 [INFO] encoded CSR
2020/09/29 17:35:22 [INFO] signed certificate with serial number 182837546809643737009256039519642110908880768521
2020/09/29 17:35:22 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@kubernetes-master-01 k8s]# cp front-proxy-c*pem /etc/kubernetes/pki/
[root@kubernetes-master-01 k8s]# ll /etc/kubernetes/pki/
total 28
-rw------- 1 root root 1679 Sep 28 21:38 ca-key.pem
-rw-r--r-- 1 root root 960 Sep 28 21:38 ca.csr
-rw-r--r-- 1 root root 1273 Sep 28 21:38 ca.pem
-rw------- 1 root root 1675 Sep 29 18:04 front-proxy-ca-key.pem
-rw-r--r-- 1 root root 1143 Sep 29 18:04 front-proxy-ca.pem
-rw------- 1 root root 1675 Sep 29 18:04 front-proxy-client-key.pem
-rw-r--r-- 1 root root 1188 Sep 29 18:04 front-proxy-client.pem
[root@kubernetes-master-01 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7b5c749bf5-l5ks2 1/1 Running 0 27h
metrics-server-78c5d9dfff-2v8fd 1/1 Running 0 12sCopy to clipboardErrorCopied
- 安装
metrics-server
[root@kubernetes-master-01 ~]# cat > components.yaml <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:aggregated-metrics-reader
labels:
rbac.authorization.k8s.io/aggregate-to-view: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1beta1.metrics.k8s.io
spec:
service:
name: metrics-server
namespace: kube-system
group: metrics.k8s.io
version: v1beta1
insecureSkipTLSVerify: true
groupPriorityMinimum: 100
versionPriority: 100
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metrics-server
namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: metrics-server
namespace: kube-system
labels:
k8s-app: metrics-server
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
- name: ca-ssl
hostPath:
path: /etc/kubernetes/pki
containers:
- name: metrics-server
image: registry.cn-hangzhou.aliyuncs.com/k8sos/metrics-server:v0.3.7
imagePullPolicy: IfNotPresent
args:
- --cert-dir=/tmp
- --secure-port=4443
- --metric-resolution=30s
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem
- --requestheader-username-headers=X-Remote-User
- --requestheader-group-headers=X-Remote-Group
- --requestheader-extra-headers-prefix=X-Remote-Extra-
ports:
- name: main-port
containerPort: 4443
protocol: TCP
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- name: tmp-dir
mountPath: /tmp
- name: ca-ssl
mountPath: /etc/kubernetes/pki
nodeSelector:
kubernetes.io/os: linux
---
apiVersion: v1
kind: Service
metadata:
name: metrics-server
namespace: kube-system
labels:
kubernetes.io/name: "Metrics-server"
kubernetes.io/cluster-service: "true"
spec:
selector:
k8s-app: metrics-server
ports:
- port: 443
protocol: TCP
targetPort: main-port
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
- nodes/stats
- namespaces
- configmaps
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
EOF
[root@kubernetes-master-01 ~]# kubectl delete -f components.yaml
clusterrole.rbac.authorization.k8s.io "system:aggregated-metrics-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "metrics-server:system:auth-delegator" deleted
rolebinding.rbac.authorization.k8s.io "metrics-server-auth-reader" deleted
apiservice.apiregistration.k8s.io "v1beta1.metrics.k8s.io" deleted
serviceaccount "metrics-server" deleted
deployment.apps "metrics-server" deleted
service "metrics-server" deleted
clusterrole.rbac.authorization.k8s.io "system:metrics-server" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:metrics-server" deletedCopy to clipboardErrorCopied
- 在
kube-apiserver配置中加入metrics-server证书
# /etc/kubernetes/cfg/kube-apiserver.conf
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem Copy to clipboardErrorCopied
- 验证
[root@kubernetes-master-01 ~]# kubectl top pods -n kube-system
NAME CPU(cores) MEMORY(bytes)
coredns-7b5c749bf5-l5ks2 3m 15Mi
metrics-server-589847c86f-cg6d4 1m 11MiCopy to clipboardErrorCopied
创建HPA
HPA是根据CPU的使用率、内存使用率自动扩展Pod数量的，所以要使用HPA就必须定义Requests参数
kind: Deployment
apiVersion: apps/v1
metadata:
name: nginx
namespace: default
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
resources:
limits:
cpu: 10m
memory: 50Mi
requests:
cpu: 10m
memory: 50Mi
---
kind: Service
apiVersion: v1
metadata:
name: svc
namespace: default
spec:
selector:
app: nginx
ports:
- port: 80
targetPort: 80
---
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: docs
namespace: default
spec:
maxReplicas: 10
minReplicas: 1
scaleTargetRef:
kind: Deployment
name: nginx
apiVersion: apps/v1
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 5Copy to clipboardErrorCopied
- 查看
[root@kubernetes-master-01 ~]# kubectl get -f hpa.yaml
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 1/1 1 1 20m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/svc ClusterIP 10.96.119.88 <none> 80/TCP 20m
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/docs Deployment/nginx 0%/5% 1 10 1 20mCopy to clipboardErrorCopied
- 加压
[root@kubernetes-master-01 ~]# ab -c 1000 -n 1000 http://10.96.119.88/index.html
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 10.96.119.88 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software: nginx/1.19.2
Server Hostname: 10.96.119.88
Server Port: 80
Document Path: /index.html
Document Length: 612 bytes
Concurrency Level: 1000
Time taken for tests: 6.389 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 845000 bytes
HTML transferred: 612000 bytes
Requests per second: 156.51 [#/sec] (mean)
Time per request: 6389.404 [ms] (mean)
Time per request: 6.389 [ms] (mean, across all concurrent requests)
Transfer rate: 129.15 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 230 749.0 1 3051
Processing: 30 879 738.9 700 3956
Waiting: 0 875 739.0 700 3955
Total: 34 1108 1351.2 701 6378
Percentage of the requests served within a certain time (ms)
50% 701
66% 799
75% 800
80% 895
90% 2800
95% 5150
98% 5349
99% 6378
100% 6378 (longest request)Copy to clipboardErrorCopied
- 查看结果
[root@kubernetes-master-01 ~]# kubectl get pods -w -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-6cf7488b57-992qk 0/1 ContainerCreating 0 43s <none> kubernetes-master-01 <none> <none>
nginx-6cf7488b57-bd887 1/1 Running 0 21m 10.243.24.2 kubernetes-master-01 <none> <none>
nginx-6cf7488b57-h8gk6 1/1 Running 0 58s 10.243.24.14 kubernetes-master-01 <none> <none>
nginx-6cf7488b57-hfr7b 1/1 Running 0 58s 10.243.24.12 kubernetes-master-01 <none> <none>
nginx-6cf7488b57-sbtz9 1/1 Running 0 59s 10.243.24.13 kubernetes-master-01 <none> <none>
nginx-6cf7488b57-xg8sb 0/1 ContainerCreating 0 43s <none> kubernetes-master-01 <none> <none>
nginx-6cf7488b57-992qk 1/1 Running 0 43s 10.243.24.16 kubernetes-master-01 <none> <none>
[root@kubernetes-master-01 ~]# kubectl get hpa -w
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
docs Deployment/nginx 0%/5% 1 10 6 22m
docs Deployment/nginx 5%/5% 1 10 6 22m
docs Deployment/nginx 11%/5% 1 10 6 23m
docs Deployment/nginx 45%/5% 1 10 10 23mCopy to clipboardErrorCopied