• k8s之自定义指标API部署prometheus


    1.自定义指标-prometheus

    node_exporter是agent;PromQL相当于sql语句来查询数据;

    k8s-prometheus-adapter:prometheus是不能直接解析k8s的指标的,需要借助k8s-prometheus-adapter转换成api;

    kube-state-metrics是用来整合数据的.

    访问:https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/prometheus

    git clone https://github.com/iKubernetes/k8s-prom.git
    cd k8s-prom && kubectl apply -f namespace.yaml
    # 部署node_exporter
    cd node_exporter/ && kubectl apply -f .
    # 部署prometheus,注释掉资源限制limit,
    cd prometheus/ && vim prometheus-deploy.yaml && kubectl apply -f .
    #resources:
    #  limits:
    #    memory: 200Mi
    这个pod没有部署好,prometheus就无法收集到数据,导致grafana界面没有数据,浪费了一天时间
    kubectl get pods -n prom
    prometheus-server-64877844d4-gx4jr 1/1 Running 0 <invalid>
    

    访问NodePort,访问prometheus

    部署k8s-prometheus-adapter,需要自制证书

    cd kube-state-metrics/ && kubectl apply -f .
    cd /etc/kubernetes/pki/
    (umask 077; openssl genrsa -out serving.key 2048)
    openssl req -new -key serving.key -out serving.csr -subj "/CN=serving"
    openssl  x509 -req -in serving.csr -CA ./ca.crt -CAkey ./ca.key -CAcreateserial -out serving.crt -days 3650
    # custom-metrics-apiserver-deployment.yaml会用到secretName: cm-adapter-serving-certs
    kubectl create secret generic cm-adapter-serving-certs --from-file=serving.crt=./serving.crt --from-file=serving.key=./serving.key  -n prom
    
    # 部署k8s-prometheus-adapter,由于版本问题,需要下载两个文件,将两个文件中的名称空间改为prom
    cd k8s-prometheus-adapter/
    mv custom-metrics-apiserver-deployment.yaml ..
    wget https://raw.githubusercontent.com/DirectXMan12/k8s-prometheus-adapter/master/deploy/manifests/custom-metrics-apiserver-deployment.yam
    wget https://raw.githubusercontent.com/DirectXMan12/k8s-prometheus-adapter/master/deploy/manifests/custom-metrics-config-map.yaml
    kubectl apply -f .
    
    kubectl api-versions  # 必须出现这个api,并且开启代理可以访问到数据
    custom.metrics.k8s.io/v1beta1
    kubectl proxy --port=8080
    curl http://localhost:8080/apis/custom.metrics.k8s.io/v1beta1/
    # prometheus和grafana整合
    wget https://raw.githubusercontent.com/kubernetes-retired/heapster/master/deploy/kube-config/influxdb/grafana.yaml
    把namespace: kube-system改成prom,有两处;
    把env里面的下面两个注释掉:
    - name: INFLUXDB_HOST
     value: monitoring-influxdb
    在最有一行加个type: NodePort
     ports:
      - port: 80
        targetPort: 3000
      selector:
        k8s-app: grafana
      type: NodePort
    kubectl apply -f grafana.yaml
    kubectl get svc -n prom
    monitoring-grafana  NodePort    10.96.228.0      <none>  80:30336/TCP     13h
    

    prom名称空间内的所有pod

    访问:10.0.0.20:30336

    两个k8s模板:https://grafana.com/dashboards/6417 https://grafana.com/dashboards/315

    一切顺利的话,立马就能看到监控数据

    2.HPA(水平pod自动扩展)

    当pod压力大了,会根据负载自动扩展Pod个数以缓解压力

    kubectl api-versions |grep auto
    创建一个带有资源限制的pod
    kubectl run myapp --image=ikubernetes/myapp:v1 --replicas=1 
    --requests='cpu=50m,memory=256Mi' --limits='cpu=50m,memory=256Mi' 
    --labels='app=myapp' --expose --port=80
    # 让myapp这个控制器支持自动扩展,--cpu-percent表示cpu超过这个值就开始扩展
    kubectl autoscale deployment myapp --min=1 --max=5 --cpu-percent=60
    kubectl get hpa
    # 对pod进行压力测试
    kubectl patch svc myapp -p '{"spec":{"type": "NodePort"}}'
    yum install httpd-tools
    # 随着cpu压力的上升,会看到自动扩展为4个或更多的pod
    ab -c 1000 -n 5000000 http://172.16.1.100:31990/index.html
    # hpa v1版本只能根据cpu利用率扩展pod,hpa v2可以根据自定义指标利用率水平扩展pod
    kubectl delete hpa myapp
    
    cat hpa-v2-demo.yaml 
    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    metadata:
      name: myapp-hpa-v2
    spec:
      scaleTargetRef: # 根据什么指标来做评估压力
        apiVersion: apps/v1 
        kind: Deployment
        name: myapp  # 对哪个控制器做自动扩展
      minReplicas: 1
      maxReplicas: 10
      metrics: # 依据哪些指标来进行评估
      - type: Resource # 基于资源进行评估
        resource: 
          name: cpu
          targetAverageUtilization: 55 # cpu使用率超过55%,就自动水平扩展pod个数
      - type: Resource
        resource:
          name: memory  # v2版可以根据内存进行评估
          targetAverageValue: 50Mi # 内存使用超过50M,就自动水平扩展pod个数
    kubectl apply -f hpa-v2-demo.yaml
    # 进行压测即可看到pod会自动扩展
    # 自定义的资源指标,pod被开发好之后,得支持这些指标,否则就是白写
    # 下面这个例子中支持并发参数的镜像地址:https://hub.docker.com/r/ikubernetes/metrics-app/
    cat hpa-v2-custom.yaml 
    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    metadata:
      name: myapp-hpa-v2
    spec:
      scaleTargetRef:
        apiVersion: apps/v1
        kind: Deployment
        name: myapp
      minReplicas: 1
      maxReplicas: 10
      metrics:
      - type: Pods  # 利用pod中定义的指标进行扩缩
        pods: 
          metricName: http_requests  # 自定义的资源指标
            targetAverageValue: 800m # m表示个数,并发数800
    

    参考博客:http://blog.itpub.net/28916011/viewspace-2216340/

    prometheus监控mysql、k8s:https://www.cnblogs.com/sfnz/p/6566951.html

  • 相关阅读:
    指针问题,p+i什么意思i代表什么
    怎么用C语言写一个飞机程序
    switch()语句中()中必须是常量吗
    一元二次方程运行结果中输入上系数后总显示输入不合法无论系数可以得出实根
    我想学号图论求大神请教
    c-freelib库
    十进制转十六进制问题(有代码)
    关于逐个读取同一目录下txt文本的问题
    JAVA-JDK1.7-ConCurrentHashMap-测试和验证
    JAVA-JDK1.7-ConCurrentHashMap-源码并且debug说明
  • 原文地址:https://www.cnblogs.com/fawaikuangtu123/p/11296540.html
Copyright © 2020-2023  润新知