HPA（Horizontal Pod Autoscaler）在k8s集群中用于POD水平自动伸缩，它是基于CPU和内存利用率对Deployment和Replicaset控制器中的pod数量进行自动扩缩容（除了CPU和内存利用率之外，也可以基于其他应程序提供的度量指标custom metrics进行自动扩缩容）。pod自动缩放不适用于无法缩放的对象，比如DaemonSets。HPA由Kubernetes API资源和控制器实现。资源决定了控制器的行为，控制器会周期性的获取CPU和内存利用率，并与目标值相比较后来调整replication controller或deployment中的副本数量。
HPA使用前提条件：
- 集群中部署了Metrics Server插件, 可以获取到Pod的CPU和内存利用率。
- Pod部署的Yaml文件中必须设置资源限制和资源请求。
以下对K8S集群使用HPA进行Pod自动伸缩做个测试记录
1. 编译一个测试容器
[root@k8s-master01 work]# cat kevin-t.yaml
apiVersion: v1
kind: Service
metadata:
name: kevin-t
namespace: default
labels:
app: kevin-t
spec:
type: NodePort
selector:
app: kevin-t
ports:
- name: http
port: 8088
targetPort: 80
nodePort: 30888
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kevin-t
namespace: default
spec:
replicas: 1
minReadySeconds: 10
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
selector:
matchLabels:
app: kevin-t
template:
metadata:
labels:
app: kevin-t
spec:
containers:
- name: kevin-t
image: nginx:1.7.9
imagePullPolicy: IfNotPresent
ports:
- name: kport
containerPort: 80
resources:
requests:
cpu: 100m
memory: 50Mi
limits:
cpu: 100m
memory: 50Mi
lifecycle:
postStart:
exec:
command: ["/bin/sh","-c","touch /tmp/health"]
livenessProbe:
exec:
command: ["test","-e","/tmp/health"]
initialDelaySeconds: 5
timeoutSeconds: 3
periodSeconds: 5
readinessProbe:
tcpSocket:
port: kport
initialDelaySeconds: 5
timeoutSeconds: 3
periodSeconds: 5
执行创建
[root@k8s-master01 work]# kubectl apply -f kevin-t.yaml
查看pod
[root@k8s-master01 work]# kubectl get pods|grep kevin-t
kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 119s
[root@k8s-master01 work]# kubectl get svc|grep kevin-t
kevin-t NodePort 10.254.120.80 <none> 8088:30888/TCP 4m20s
访问
[root@k8s-master01 work]# curl http://172.16.60.235:30888
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
查看metrcis-service采集到的该pod的cpu和内存数据
[root@k8s-master01 work]# kubectl top pods
NAME CPU(cores) MEMORY(bytes)
dnsutils-ds-22q87 0m 0Mi
dnsutils-ds-jcvng 0m 0Mi
dnsutils-ds-zjg5l 0m 0Mi
kevin-t-66cdc9cdbf-jwvnv 1m 3Mi
my-nginx-765d6dcff7-gs97l 0m 3Mi
my-nginx-765d6dcff7-lbjm6 0m 2Mi
nginx-ds-98rm2 0m 2Mi
nginx-ds-bbx68 0m 2Mi
nginx-ds-ngqcm 0m 2Mi
通过上面可知，使用kevin-t的deployment控制的kevin-t的pod容器资源请求和资源限制都是100m的CPU（即0.1核）和50M的内存。当前该pod所用CPU是1m，内存是3M
2. 创建HPA限制
设置kevin-t的 deployment的最大最小副本数，HPA对应pod的CPU和内存指标做限制。
参数解释：
scaleTargetRef: 要缩放的目标资源
metrics: 计算所需Pod副本数量的指标列表，每个指标单独计算，取所有计算结果的最大值作为最终副本数量
- external 引用非附属于任何对象的全局指标，可以是集群之外的组件的指标数据，如消息队列长度
- object 引用描述集群中某单一对象的特定指标，如Ingress对象上的hits-per-second等
- pods 引用当前被弹性伸缩的Pod对象的特定指标
- resource 引用资源指标，即当前被弹性伸缩的Pod对象中容器的requests和limits中定义的指标
- type 指标源的类型，可为Objects、Pods、Resource
另外注意两个指标阈值参数：
- targetAverageUtilization 表示的是百分比
- targetAverageValue 表示的是数值，比如100m的CPU、100Mi的内存
[root@k8s-master01 work]# cat kevin-t-hap.yml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: kevin-t-hap
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: kevin-t
minReplicas: 1
maxReplicas: 3
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 80
- type: Resource
resource:
name: memory
targetAverageValue: 30Mi
如上，设置了kevin-t的deployment控制的pod的HPA限制，当cpu使用超过设置的80%，内存使用超过30Mi时就触发自动扩容，副本数最小为1，最大为3。
查看创建的HPA
[root@k8s-master01 work]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
kevin-t-hap Deployment/kevin-t 3194880/30Mi, 1%/80% 1 3 3 2m15s
[root@k8s-master01 work]# kubectl describe hpa kevin-t-hap
Name: kevin-t-hap
Namespace: default
Labels: <none>
Annotations: CreationTimestamp: Fri, 14 Aug 2020 17:00:01 +0800
Reference: Deployment/kevin-t
Metrics: ( current / target )
resource memory on pods: 3291818666m / 30Mi
resource cpu on pods (as a percentage of request): 1% (1m) / 80%
Min replicas: 1
Max replicas: 3
Deployment pods: 3 current / 3 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
ScalingLimited True TooManyReplicas the desired replica count is more than the maximum replica count
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 4m3s horizontal-pod-autoscaler New size: 3; reaso
3. 压力测试，验证HPA的扩缩容
使用ab命令进行压力测试，主要有两个参数：
-n ：总共的请求执行数，缺省是1；
-c： 并发数，缺省是1；
测试之前kevin-t的pod是一个副本数 [root@k8s-master01 work]# kubectl get pods|grep kevin-t kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 119s 接下来进行压力测试 [root@k8s-master01 work]# ab -c 1000 -n 500000000000 http://172.16.60.235:30888/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 172.16.60.235 (be patient) apr_socket_recv: Connection reset by peer (104) Total of 12623 requests completed 发现此时，kevin-t 已经扩到2个副本了 [root@k8s-master02 ~]# kubectl get pods|grep kevin-t kevin-t-66cdc9cdbf-5h564 1/1 Running 0 38s kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 35m 再加大压力测试： [root@k8s-master01 work]# ab -c 100 -n 50000000000000000 http://172.16.60.235:30888/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 172.16.60.235 (be patient) 在上面压力测试中，另外打开一个终端，发现kevin-t已经扩容到最大的3个副本了！ [root@k8s-master02 ~]# kubectl get pods|grep kevin-t kevin-t-66cdc9cdbf-5h564 1/1 Running 0 4m52s kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 39m kevin-t-66cdc9cdbf-zpbfs 1/1 Running 0 80s 查看hpa，实时查看pod的cpu和内存状态 [root@k8s-master01 work]# kubectl get hpa -w NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE kevin-t-hap Deployment/kevin-t 4434602666m/30Mi, 1%/80% 1 3 3 19m kevin-t-hap Deployment/kevin-t 4405930666m/30Mi, 1%/80% 1 3 3 19m kevin-t-hap Deployment/kevin-t 4441429333m/30Mi, 1%/80% 1 3 3 20m kevin-t-hap Deployment/kevin-t 4441429333m/30Mi, 1%/80% 1 3 3 20m kevin-t-hap Deployment/kevin-t 4945920/30Mi, 1%/80% 1 3 2 20m kevin-t-hap Deployment/kevin-t 4945920/30Mi, 1%/80% 1 3 2 21m ............... 观察kevin-t的pod情况 [root@k8s-master02 ~]# kubectl get pods|grep kevin-t -w kevin-t-66cdc9cdbf-5h564 1/1 Running 0 11m kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 46m kevin-t-66cdc9cdbf-zpbfs 1/1 Running 0 7m36s [root@k8s-master02 ~]# kubectl get pods|grep kevin-t -w kevin-t-66cdc9cdbf-5h564 1/1 Terminating 0 12m kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 47m 发现压力测试过一段时间后，随着pod的cpu和内存使用数值下降，副本数也在逐步减少到最小的1个副本 [root@k8s-master02 ~]# kubectl get pods|grep kevin-t kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 49m