• k8s配置alertmanager发送报警到qq邮箱


    k8s配置alertmanager发送报警到qq邮箱

    一、Prometheus报警处理流程

    1)Prometheus Server监控目标主机上暴露的http接口(这里假设接口A),通过Promethes配置的'scrape_interval'定义的时间间隔,定期采集目标主机上监控数据。

    2)当接口A不可用的时候,Server端会持续的尝试从接口中取数据,直到"scrape_timeout"时间后停止尝试。这时候把接口的状态变为“DOWN”。

    3)Prometheus同时根据配置的"evaluation_interval"的时间间隔,定期(默认1min)的对Alert Rule进行评估;当到达评估周期的时候,发现接口A为DOWN,即UP=0为真,激活Alert,进入“PENDING”状态,并记录当前active的时间;

    4)当下一个alert rule的评估周期到来的时候,发现UP=0继续为真,然后判断警报Active的时间是否已经超出rule里的‘for’ 持续时间,如果未超出,则进入下一个评估周期;如果时间超出,则alert的状态变为“FIRING”;同时调用Alertmanager接口,发送相关报警数据。

    5)AlertManager收到报警数据后,会将警报信息进行分组,然后根据alertmanager配置的“group_wait”时间先进行等待。等wait时间过后再发送报警信息。

    6)属于同一个Alert Group的警报,在等待的过程中可能进入新的alert,如果之前的报警已经成功发出,那么间隔“group_interval”的时间间隔后再重新发送报警信息。比如配置的是邮件报警,那么同属一个group的报警信息会汇总在一个邮件里进行发送。

    7)如果Alert Group里的警报一直没发生变化并且已经成功发送,等待‘repeat_interval’时间间隔之后再重复发送相同的报警邮件;如果之前的警报没有成功发送,则相当于触发第6条条件,则需要等待group_interval时间间隔后重复发送。

    8)同时最后至于警报信息具体发给谁,满足什么样的条件下指定警报接收人,设置不同报警发送频率,这里有alertmanager的route路由规则进行配置。

    二、Prometheus及Alertmanager配置

    2.1、配置alertmanager及告警规则

    1)创建alertmanager配置文件

    [root@k8s-master1 prometheus]# cat alertmanager-cm.yaml
    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: alertmanager
      namespace: monitor-sa
    data:
      alertmanager.yml: |-
        global:
          resolve_timeout: 1m
          smtp_smarthost: 'smtp.163.com:25'
          smtp_from: '18665870472@163.com'
          smtp_auth_username: '18665870472'
          smtp_auth_password: 'GGCTEDQDVLKPCIID'
          smtp_require_tls: false
        route:	#用于设置告警的分发策略
          group_by: [alertname]	# 采用哪个标签来作为分组依据
          group_wait: 10s	# 组告警等待时间。也就是告警产生后等待10s,如果有同组告警一起发出
          group_interval: 10s	# 上下两组发送告警的间隔时间
          repeat_interval: 10m	# 重复发送告警的时间,减少相同邮件的发送频率,默认是1h
          receiver: default-receiver	#定义谁来收告警
        receivers:
        - name: 'default-receiver'
          email_configs:
          - to: '352972405@qq.com'
            send_resolved: true
            
    [root@k8s-master1 prometheus]# kubectl apply -f alertmanager-cm.yaml
    configmap/alertmanager created
    

    2)创建prometheus和告警规则配置文件

    [root@k8s-master1 prometheus]# cat prometheus-alertmanager-cfg.yaml
    kind: ConfigMap
    apiVersion: v1
    metadata:
      labels:
        app: prometheus
      name: prometheus-config
      namespace: monitor-sa
    data:
      prometheus.yml: |
        rule_files:
        - /etc/prometheus/rules.yml
        alerting:
          alertmanagers:
          - static_configs:
            - targets: ["localhost:9093"]
        global:
          scrape_interval: 15s
          scrape_timeout: 10s
          evaluation_interval: 1m
        scrape_configs:
        - job_name: 'kubernetes-node'
          kubernetes_sd_configs:
          - role: node
          relabel_configs:
          - source_labels: [__address__]
            regex: '(.*):10250'
            replacement: '${1}:9100'
            target_label: __address__
            action: replace
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
        - job_name: 'kubernetes-node-cadvisor'
          kubernetes_sd_configs:
          - role:  node
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
        - job_name: 'kubernetes-apiserver'
          kubernetes_sd_configs:
          - role: endpoints
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          relabel_configs:
          - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
            action: keep
            regex: default;kubernetes;https
        - job_name: 'kubernetes-service-endpoints'
          kubernetes_sd_configs:
          - role: endpoints
          relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
            action: replace
            target_label: __scheme__
            regex: (https?)
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
            action: replace
            target_label: __address__
            regex: ([^:]+)(?::d+)?;(d+)
            replacement: $1:$2
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            action: replace
            target_label: kubernetes_name 
        - job_name: 'kubernetes-pods'
          kubernetes_sd_configs:
          - role: pod
          relabel_configs:
          - action: keep
            regex: true
            source_labels:
            - __meta_kubernetes_pod_annotation_prometheus_io_scrape
          - action: replace
            regex: (.+)
            source_labels:
            - __meta_kubernetes_pod_annotation_prometheus_io_path
            target_label: __metrics_path__
          - action: replace
            regex: ([^:]+)(?::d+)?;(d+)
            replacement: $1:$2
            source_labels:
            - __address__
            - __meta_kubernetes_pod_annotation_prometheus_io_port
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - action: replace
            source_labels:
            - __meta_kubernetes_namespace
            target_label: kubernetes_namespace
          - action: replace
            source_labels:
            - __meta_kubernetes_pod_name
            target_label: kubernetes_pod_name
        - job_name: 'kubernetes-schedule'
          scrape_interval: 5s
          static_configs:
          - targets: ['192.168.40.180:10251']
        - job_name: 'kubernetes-controller-manager'
          scrape_interval: 5s
          static_configs:
          - targets: ['192.168.40.180:10252']
        - job_name: 'kubernetes-kube-proxy'
          scrape_interval: 5s
          static_configs:
          - targets: ['192.168.40.180:10249','192.168.40.181:10249','192.168.40.182:10249']
        - job_name: 'kubernetes-etcd'
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/k8s-certs/etcd/ca.crt
            cert_file: /var/run/secrets/kubernetes.io/k8s-certs/etcd/server.crt
            key_file: /var/run/secrets/kubernetes.io/k8s-certs/etcd/server.key
          scrape_interval: 5s
          static_configs:
          - targets: ['192.168.40.180:2379']
      rules.yml: |
        groups:
        - name: example
          rules:
          - alert: kube-proxy的cpu使用率大于80%
            expr: rate(process_cpu_seconds_total{job=~"kubernetes-kube-proxy"}[1m]) * 100 > 80
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}组件的cpu使用率超过80%"
          - alert:  kube-proxy的cpu使用率大于90%
            expr: rate(process_cpu_seconds_total{job=~"kubernetes-kube-proxy"}[1m]) * 100 > 90
            for: 2s
            labels:
              severity: critical
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}组件的cpu使用率超过90%"
          - alert: scheduler的cpu使用率大于80%
            expr: rate(process_cpu_seconds_total{job=~"kubernetes-schedule"}[1m]) * 100 > 80
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}组件的cpu使用率超过80%"
          - alert:  scheduler的cpu使用率大于90%
            expr: rate(process_cpu_seconds_total{job=~"kubernetes-schedule"}[1m]) * 100 > 90
            for: 2s
            labels:
              severity: critical
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}组件的cpu使用率超过90%"
          - alert: controller-manager的cpu使用率大于80%
            expr: rate(process_cpu_seconds_total{job=~"kubernetes-controller-manager"}[1m]) * 100 > 80
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}组件的cpu使用率超过80%"
          - alert:  controller-manager的cpu使用率大于90%
            expr: rate(process_cpu_seconds_total{job=~"kubernetes-controller-manager"}[1m]) * 100 > 0
            for: 2s
            labels:
              severity: critical
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}组件的cpu使用率超过90%"
          - alert: apiserver的cpu使用率大于80%
            expr: rate(process_cpu_seconds_total{job=~"kubernetes-apiserver"}[1m]) * 100 > 80
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}组件的cpu使用率超过80%"
          - alert:  apiserver的cpu使用率大于90%
            expr: rate(process_cpu_seconds_total{job=~"kubernetes-apiserver"}[1m]) * 100 > 90
            for: 2s
            labels:
              severity: critical
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}组件的cpu使用率超过90%"
          - alert: etcd的cpu使用率大于80%
            expr: rate(process_cpu_seconds_total{job=~"kubernetes-etcd"}[1m]) * 100 > 80
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}组件的cpu使用率超过80%"
          - alert:  etcd的cpu使用率大于90%
            expr: rate(process_cpu_seconds_total{job=~"kubernetes-etcd"}[1m]) * 100 > 90
            for: 2s
            labels:
              severity: critical
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}组件的cpu使用率超过90%"
          - alert: kube-state-metrics的cpu使用率大于80%
            expr: rate(process_cpu_seconds_total{k8s_app=~"kube-state-metrics"}[1m]) * 100 > 80
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "{{$labels.instance}}的{{$labels.k8s_app}}组件的cpu使用率超过80%"
              value: "{{ $value }}%"
              threshold: "80%"      
          - alert: kube-state-metrics的cpu使用率大于90%
            expr: rate(process_cpu_seconds_total{k8s_app=~"kube-state-metrics"}[1m]) * 100 > 0
            for: 2s
            labels:
              severity: critical
            annotations:
              description: "{{$labels.instance}}的{{$labels.k8s_app}}组件的cpu使用率超过90%"
              value: "{{ $value }}%"
              threshold: "90%"      
          - alert: coredns的cpu使用率大于80%
            expr: rate(process_cpu_seconds_total{k8s_app=~"kube-dns"}[1m]) * 100 > 80
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "{{$labels.instance}}的{{$labels.k8s_app}}组件的cpu使用率超过80%"
              value: "{{ $value }}%"
              threshold: "80%"      
          - alert: coredns的cpu使用率大于90%
            expr: rate(process_cpu_seconds_total{k8s_app=~"kube-dns"}[1m]) * 100 > 90
            for: 2s
            labels:
              severity: critical
            annotations:
              description: "{{$labels.instance}}的{{$labels.k8s_app}}组件的cpu使用率超过90%"
              value: "{{ $value }}%"
              threshold: "90%"      
          - alert: kube-proxy打开句柄数>600
            expr: process_open_fds{job=~"kubernetes-kube-proxy"}  > 600
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}打开句柄数>600"
              value: "{{ $value }}"
          - alert: kube-proxy打开句柄数>1000
            expr: process_open_fds{job=~"kubernetes-kube-proxy"}  > 1000
            for: 2s
            labels:
              severity: critical
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}打开句柄数>1000"
              value: "{{ $value }}"
          - alert: kubernetes-schedule打开句柄数>600
            expr: process_open_fds{job=~"kubernetes-schedule"}  > 600
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}打开句柄数>600"
              value: "{{ $value }}"
          - alert: kubernetes-schedule打开句柄数>1000
            expr: process_open_fds{job=~"kubernetes-schedule"}  > 1000
            for: 2s
            labels:
              severity: critical
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}打开句柄数>1000"
              value: "{{ $value }}"
          - alert: kubernetes-controller-manager打开句柄数>600
            expr: process_open_fds{job=~"kubernetes-controller-manager"}  > 600
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}打开句柄数>600"
              value: "{{ $value }}"
          - alert: kubernetes-controller-manager打开句柄数>1000
            expr: process_open_fds{job=~"kubernetes-controller-manager"}  > 1000
            for: 2s
            labels:
              severity: critical
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}打开句柄数>1000"
              value: "{{ $value }}"
          - alert: kubernetes-apiserver打开句柄数>600
            expr: process_open_fds{job=~"kubernetes-apiserver"}  > 600
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}打开句柄数>600"
              value: "{{ $value }}"
          - alert: kubernetes-apiserver打开句柄数>1000
            expr: process_open_fds{job=~"kubernetes-apiserver"}  > 1000
            for: 2s
            labels:
              severity: critical
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}打开句柄数>1000"
              value: "{{ $value }}"
          - alert: kubernetes-etcd打开句柄数>600
            expr: process_open_fds{job=~"kubernetes-etcd"}  > 600
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}打开句柄数>600"
              value: "{{ $value }}"
          - alert: kubernetes-etcd打开句柄数>1000
            expr: process_open_fds{job=~"kubernetes-etcd"}  > 1000
            for: 2s
            labels:
              severity: critical
            annotations:
              description: "{{$labels.instance}}的{{$labels.job}}打开句柄数>1000"
              value: "{{ $value }}"
          - alert: coredns
            expr: process_open_fds{k8s_app=~"kube-dns"}  > 600
            for: 2s
            labels:
              severity: warnning 
            annotations:
              description: "插件{{$labels.k8s_app}}({{$labels.instance}}): 打开句柄数超过600"
              value: "{{ $value }}"
          - alert: coredns
            expr: process_open_fds{k8s_app=~"kube-dns"}  > 1000
            for: 2s
            labels:
              severity: critical
            annotations:
              description: "插件{{$labels.k8s_app}}({{$labels.instance}}): 打开句柄数超过1000"
              value: "{{ $value }}"
          - alert: kube-proxy
            expr: process_virtual_memory_bytes{job=~"kubernetes-kube-proxy"}  > 2000000000
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "组件{{$labels.job}}({{$labels.instance}}): 使用虚拟内存超过2G"
              value: "{{ $value }}"
          - alert: scheduler
            expr: process_virtual_memory_bytes{job=~"kubernetes-schedule"}  > 2000000000
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "组件{{$labels.job}}({{$labels.instance}}): 使用虚拟内存超过2G"
              value: "{{ $value }}"
          - alert: kubernetes-controller-manager
            expr: process_virtual_memory_bytes{job=~"kubernetes-controller-manager"}  > 2000000000
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "组件{{$labels.job}}({{$labels.instance}}): 使用虚拟内存超过2G"
              value: "{{ $value }}"
          - alert: kubernetes-apiserver
            expr: process_virtual_memory_bytes{job=~"kubernetes-apiserver"}  > 2000000000
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "组件{{$labels.job}}({{$labels.instance}}): 使用虚拟内存超过2G"
              value: "{{ $value }}"
          - alert: kubernetes-etcd
            expr: process_virtual_memory_bytes{job=~"kubernetes-etcd"}  > 2000000000
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "组件{{$labels.job}}({{$labels.instance}}): 使用虚拟内存超过2G"
              value: "{{ $value }}"
          - alert: kube-dns
            expr: process_virtual_memory_bytes{k8s_app=~"kube-dns"}  > 2000000000
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "插件{{$labels.k8s_app}}({{$labels.instance}}): 使用虚拟内存超过2G"
              value: "{{ $value }}"
          - alert: HttpRequestsAvg
            expr: sum(rate(rest_client_requests_total{job=~"kubernetes-kube-proxy|kubernetes-kubelet|kubernetes-schedule|kubernetes-control-manager|kubernetes-apiservers"}[1m]))  > 1000
            for: 2s
            labels:
              team: admin
            annotations:
              description: "组件{{$labels.job}}({{$labels.instance}}): TPS超过1000"
              value: "{{ $value }}"
              threshold: "1000"   
          - alert: Pod_restarts
            expr: kube_pod_container_status_restarts_total{namespace=~"kube-system|default|monitor-sa"} > 0
            for: 2s
            labels:
              severity: warnning
            annotations:
              description: "在{{$labels.namespace}}名称空间下发现{{$labels.pod}}这个pod下的容器{{$labels.container}}被重启,这个监控指标是由{{$labels.instance}}采集的"
              value: "{{ $value }}"
              threshold: "0"
          - alert: Pod_waiting
            expr: kube_pod_container_status_waiting_reason{namespace=~"kube-system|default"} == 1
            for: 2s
            labels:
              team: admin
            annotations:
              description: "空间{{$labels.namespace}}({{$labels.instance}}): 发现{{$labels.pod}}下的{{$labels.container}}启动异常等待中"
              value: "{{ $value }}"
              threshold: "1"   
          - alert: Pod_terminated
            expr: kube_pod_container_status_terminated_reason{namespace=~"kube-system|default|monitor-sa"} == 1
            for: 2s
            labels:
              team: admin
            annotations:
              description: "空间{{$labels.namespace}}({{$labels.instance}}): 发现{{$labels.pod}}下的{{$labels.container}}被删除"
              value: "{{ $value }}"
              threshold: "1"
          - alert: Etcd_leader
            expr: etcd_server_has_leader{job="kubernetes-etcd"} == 0
            for: 2s
            labels:
              team: admin
            annotations:
              description: "组件{{$labels.job}}({{$labels.instance}}): 当前没有leader"
              value: "{{ $value }}"
              threshold: "0"
          - alert: Etcd_leader_changes
            expr: rate(etcd_server_leader_changes_seen_total{job="kubernetes-etcd"}[1m]) > 0
            for: 2s
            labels:
              team: admin
            annotations:
              description: "组件{{$labels.job}}({{$labels.instance}}): 当前leader已发生改变"
              value: "{{ $value }}"
              threshold: "0"
          - alert: Etcd_failed
            expr: rate(etcd_server_proposals_failed_total{job="kubernetes-etcd"}[1m]) > 0
            for: 2s
            labels:
              team: admin
            annotations:
              description: "组件{{$labels.job}}({{$labels.instance}}): 服务失败"
              value: "{{ $value }}"
              threshold: "0"
          - alert: Etcd_db_total_size
            expr: etcd_debugging_mvcc_db_total_size_in_bytes{job="kubernetes-etcd"} > 10000000000
            for: 2s
            labels:
              team: admin
            annotations:
              description: "组件{{$labels.job}}({{$labels.instance}}):db空间超过10G"
              value: "{{ $value }}"
              threshold: "10G"
          - alert: Endpoint_ready
            expr: kube_endpoint_address_not_ready{namespace=~"kube-system|default"} == 1
            for: 2s
            labels:
              team: admin
            annotations:
              description: "空间{{$labels.namespace}}({{$labels.instance}}): 发现{{$labels.endpoint}}不可用"
              value: "{{ $value }}"
              threshold: "1"
        - name: 物理节点状态-监控告警
          rules:
          - alert: 物理节点cpu使用率
            expr: 100-avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by(instance)*100 > 90
            for: 2s
            labels:
              severity: ccritical
            annotations:
              summary: "{{ $labels.instance }}cpu使用率过高"
              description: "{{ $labels.instance }}的cpu使用率超过90%,当前使用率[{{ $value }}],需要排查处理" 
          - alert: 物理节点内存使用率
            expr: (node_memory_MemTotal_bytes - (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes)) / node_memory_MemTotal_bytes * 100 > 90
            for: 2s
            labels:
              severity: critical
            annotations:
              summary: "{{ $labels.instance }}内存使用率过高"
              description: "{{ $labels.instance }}的内存使用率超过90%,当前使用率[{{ $value }}],需要排查处理"
          - alert: InstanceDown
            expr: up == 0
            for: 2s
            labels:
              severity: critical
            annotations:   
              summary: "{{ $labels.instance }}: 服务器宕机"
              description: "{{ $labels.instance }}: 服务器延时超过2分钟"
          - alert: 物理节点磁盘的IO性能
            expr: 100-(avg(irate(node_disk_io_time_seconds_total[1m])) by(instance)* 100) < 60
            for: 2s
            labels:
              severity: critical
            annotations:
              summary: "{{$labels.mountpoint}} 流入磁盘IO使用率过高!"
              description: "{{$labels.mountpoint }} 流入磁盘IO大于60%(目前使用:{{$value}})"
          - alert: 入网流量带宽
            expr: ((sum(rate (node_network_receive_bytes_total{device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[5m])) by (instance)) / 100) > 102400
            for: 2s
            labels:
              severity: critical
            annotations:
              summary: "{{$labels.mountpoint}} 流入网络带宽过高!"
              description: "{{$labels.mountpoint }}流入网络带宽持续5分钟高于100M. RX带宽使用率{{$value}}"
          - alert: 出网流量带宽
            expr: ((sum(rate (node_network_transmit_bytes_total{device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[5m])) by (instance)) / 100) > 102400
            for: 2s
            labels:
              severity: critical
            annotations:
              summary: "{{$labels.mountpoint}} 流出网络带宽过高!"
              description: "{{$labels.mountpoint }}流出网络带宽持续5分钟高于100M. RX带宽使用率{{$value}}"
          - alert: TCP会话
            expr: node_netstat_Tcp_CurrEstab > 1000
            for: 2s
            labels:
              severity: critical
            annotations:
              summary: "{{$labels.mountpoint}} TCP_ESTABLISHED过高!"
              description: "{{$labels.mountpoint }} TCP_ESTABLISHED大于1000%(目前使用:{{$value}}%)"
          - alert: 磁盘容量
            expr: 100-(node_filesystem_free_bytes{fstype=~"ext4|xfs"}/node_filesystem_size_bytes {fstype=~"ext4|xfs"}*100) > 80
            for: 2s
            labels:
              severity: critical
            annotations:
              summary: "{{$labels.mountpoint}} 磁盘分区使用率过高!"
              description: "{{$labels.mountpoint }} 磁盘分区使用大于80%(目前使用:{{$value}}%)"
              
    # 删除之前的配置
    [root@k8s-master1 prometheus]# kubectl delete -f prometheus-cfg.yaml
    configmap "prometheus-config" deleted
    # 更新配置
    [root@k8s-master1 prometheus]# kubectl apply -f prometheus-alertmanager-cfg.yaml
    configmap/prometheus-config created
    [root@k8s-master1 prometheus]# kubectl get cm -n monitor-sa 
    NAME                DATA   AGE
    kube-root-ca.crt    1      14h
    prometheus-config   2      29s
    

    3)安装prometheus和alertmanager

    [root@k8s-master1 prometheus]# cat prometheus-alertmanager-deploy.yaml
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: prometheus-server
      namespace: monitor-sa
      labels:
        app: prometheus
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: prometheus
          component: server
        #matchExpressions:
        #- {key: app, operator: In, values: [prometheus]}
        #- {key: component, operator: In, values: [server]}
      template:
        metadata:
          labels:
            app: prometheus
            component: server
          annotations:
            prometheus.io/scrape: 'false'
        spec:
          nodeName: k8s-node1
          serviceAccountName: monitor
          containers:
          - name: prometheus
            image: prom/prometheus:v2.2.1
            imagePullPolicy: IfNotPresent
            command:
            - "/bin/prometheus"
            args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus"
            - "--storage.tsdb.retention=24h"
            - "--web.enable-lifecycle"
            ports:
            - containerPort: 9090
              protocol: TCP
            volumeMounts:
            - mountPath: /etc/prometheus
              name: prometheus-config
            - mountPath: /prometheus/
              name: prometheus-storage-volume
            - name: k8s-certs
              mountPath: /var/run/secrets/kubernetes.io/k8s-certs/etcd/
            - name: localtime
              mountPath: /etc/localtime
          - name: alertmanager
            image: prom/alertmanager:v0.14.0
            imagePullPolicy: IfNotPresent
            args:
            - "--config.file=/etc/alertmanager/alertmanager.yml"
            - "--log.level=debug"
            ports:
            - containerPort: 9093
              protocol: TCP
              name: alertmanager
            volumeMounts:
            - name: alertmanager-config
              mountPath: /etc/alertmanager
            - name: alertmanager-storage
              mountPath: /alertmanager
            - name: localtime
              mountPath: /etc/localtime
          volumes:
            - name: prometheus-config
              configMap:
                name: prometheus-config
            - name: prometheus-storage-volume
              hostPath:
               path: /data
               type: Directory
            - name: k8s-certs
              secret:
               secretName: etcd-certs
            - name: alertmanager-config
              configMap:
                name: alertmanager
            - name: alertmanager-storage
              hostPath:
               path: /data/alertmanager
               type: DirectoryOrCreate
            - name: localtime
              hostPath:
               path: /usr/share/zoneinfo/Asia/Shanghai
               
    # 生成一个etcd-certs,这个在部署prometheus需要
    [root@k8s-master1 prometheus]# kubectl -n monitor-sa create secret generic etcd-certs --from-file=/etc/kubernetes/pki/etcd/server.key  --from-file=/etc/kubernetes/pki/etcd/server.crt --from-file=/etc/kubernetes/pki/etcd/ca.crt
    secret/etcd-certs created
    
    # 更新资源清单yaml文件
    [root@k8s-master1 prometheus]# kubectl delete -f prometheus-deploy.yaml
    deployment.apps "prometheus-server" deleted
    [root@k8s-master1 prometheus]# kubectl apply -f prometheus-alertmanager-deploy.yaml
    deployment.apps/prometheus-server created
    
    # 查看prometheus是否部署成功
    [root@k8s-master1 prometheus]# kubectl get pods -n monitor-sa | grep prometheus
    prometheus-server-76dd9f8dc6-w9fct   2/2     Running   0          32s
    

    4)部署alertmanager的service,方便在浏览器访问

    [root@k8s-master1 prometheus]# cat alertmanager-svc.yaml 
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        name: prometheus
        kubernetes.io/cluster-service: 'true'
      name: alertmanager
      namespace: monitor-sa
    spec:
      ports:
      - name: alertmanager
        nodePort: 30066
        port: 9093
        protocol: TCP
        targetPort: 9093
      selector:
        app: prometheus
      sessionAffinity: None
      type: NodePort
      
    [root@k8s-master1 prometheus]# kubectl apply -f alertmanager-svc.yaml
    service/alertmanager created
    
    # 查看service在物理机映射的端口
    [root@k8s-master1 prometheus]# kubectl get svc -n monitor-sa
    NAME           TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
    alertmanager   NodePort   10.102.118.253   <none>        9093:30066/TCP   41s
    prometheus     NodePort   10.99.104.223    <none>        9090:32367/TCP   13h
    # 注意:上面可以看到prometheus的service在物理机映射的端口是32367,alertmanager的service在物理机映射的端口是30066
    
    # 查看service在物理机映射的端口: http://192.168.40.180:30066/#/alerts
    

    image-20210712104342984

    查看接收到的邮件报警:

    image-20210712104603045

    查看prometheus的targets:

    image-20210712104845051

    2.2、监控kube-scheduler

    # 修改kube-scheduler的配置文件
    [root@k8s-master1 prometheus]# vim /etc/kubernetes/manifests/kube-scheduler.yaml
    
    # 修改如下内容
    1)把--bind-address=127.0.0.1变成--bind-address=192.168.40.180 #192.168.40.180是k8s的控制节点k8s-master1的ip
    2)把httpGet:字段下的hosts由127.0.0.1变成192.168.40.180(有两处)
    3)把—port=0删除
    
    # 重启各个节点的kubelet
    [root@k8s-node1 ~]# systemctl restart kubelet
    [root@k8s-node2 ~]# systemctl restart kubelet
    
    # 相应的端口已经被物理机监听了
    [root@k8s-master1 prometheus]# ss -antulp | grep :10251	
    tcp    LISTEN     0      128      :::10251                :::*                   users:(("kube-scheduler",pid=36945,fd=7))
    

    image-20210712105711900

    2.3、监控kube-controller-manager

    # 修改kube-scheduler的配置文件
    [root@k8s-master1 prometheus]# vim /etc/kubernetes/manifests/kube-controller-manager.yaml
    
    # 修改如下内容
    1)把--bind-address=127.0.0.1变成--bind-address=192.168.40.180 #192.168.40.180是k8s的控制节点k8s-master1的ip
    2)把httpGet:字段下的hosts由127.0.0.1变成192.168.40.180(有两处)
    3)把—port=0删除
    
    # 重启各个节点的kubelet
    [root@k8s-node1 ~]# systemctl restart kubelet
    [root@k8s-node2 ~]# systemctl restart kubelet
    
    # 查看状态
    [root@k8s-master1 prometheus]# kubectl get cs 
    Warning: v1 ComponentStatus is deprecated in v1.19+
    NAME                 STATUS    MESSAGE             ERROR
    scheduler            Healthy   ok                  
    controller-manager   Healthy   ok                  
    etcd-0               Healthy   {"health":"true"}
    
    [root@k8s-master1 prometheus]# ss -antulp | grep :10252
    tcp    LISTEN     0      128      :::10252                :::*                   users:(("kube-controller",pid=41653,fd=7))
    

    image-20210712105949370

    2.4、监控kube-proxy

    # 因为kube-proxy默认端口10249是监听在127.0.0.1上的,需要改成监听到物理节点上,按如下方法修改,线上建议在安装k8s的时候就做修改,这样风险小一些
    
    # 修改metricsBindAddress
    [root@k8s-master1 prometheus]# kubectl edit configmap kube-proxy -n kube-system
    metricsBindAddress: "0.0.0.0:10249"
    
    # 重新启动kube-proxy
    [root@k8s-master1 prometheus]# kubectl get pods -n kube-system | grep kube-proxy |awk '{print $1}' | xargs kubectl delete pods -n kube-system
    
    [root@k8s-master1 prometheus]# ss  -antulp |grep :10249
    tcp    LISTEN     0      128      :::10249                :::*                   users:(("kube-proxy",pid=45896,fd=19))
    

    image-20210712110543869

    image-20210712110601906

    2.5、alert查看

    image-20210712110705345

    image-20210712110742586

    FIRING表示prometheus已经将告警发给alertmanager,在Alertmanager 中可以看到有一个 alert。 登录到alertmanager web界面,浏览器输入192.168.40.180:30066,显示如下

    image-20210712110835355

    2.6、配置文件更新

    # 修改prometheus任何一个配置文件之后,可通过kubectl apply使配置生效,执行顺序如下:
    # 注意:生产不要这样做
    kubectl delete -f alertmanager-cm.yaml
    kubectl apply -f alertmanager-cm.yaml
    kubectl delete -f prometheus-alertmanager-cfg.yaml
    kubectl apply  -f prometheus-alertmanager-cfg.yaml 
    kubectl delete -f  prometheus-alertmanager-deploy.yaml
    kubectl apply  -f prometheus-alertmanager-deploy.yaml
    
    作者:Lawrence

    -------------------------------------------

    个性签名:独学而无友,则孤陋而寡闻。做一个灵魂有趣的人!

    扫描上面二维码关注我
    如果你真心觉得文章写得不错,而且对你有所帮助,那就不妨帮忙“推荐"一下,您的“推荐”和”打赏“将是我最大的写作动力!
    本文版权归作者所有,欢迎转载,但未经作者同意必须保留此段声明,且在文章页面明显位置给出原文连接.
  • 相关阅读:
    ES数据-MySql处理Date类型的数据导入处理
    es搜索引擎
    Java Web学习路线
    九樱天下-观察期
    一些平常的总结
    ES搜索引擎-简单入门
    工作中异常的一些总结
    抽象和封装的一点总结
    python之路_django分页及session介绍
    python之路_django中母板及cookie介绍
  • 原文地址:https://www.cnblogs.com/hujinzhong/p/15001210.html
Copyright © 2020-2023  润新知