Kube-prometheus监控jmx指标

一、程序包说明：

1)       监控代理程序包：AppMonitor.jar

2)       项目监控依赖包：AppMonitor-1.0.0.jar

3)       监控插件包：jmx_prometheus_javaagent-0.13.0.jar

二、部署说明：

1)       新建共享目录，用于存放监控代理程序包、监控插件包、以及配置文件

---
startDelaySeconds: 0
jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:8012/jmxrmi
lowercaseOutputName: false
lowercaseOutputLabelNames: false
ssl: false
username: jmx
password: jmx
rules:
- pattern: ".*"

2)       新建K8S持久化资源(PV-PVC)，用于共享上述目录到各个容器化项目中

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jmxprometheus
spec:
  capacity:
    storage: 100M
  accessModes: ["ReadWriteMany"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs
  nfs:
    path: /data/nfsprovisioner/jmxprometheus/
    server: 10.20.30.108
    
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jmxprometheus
  namespace: default
spec:
  accessModes: ["ReadWriteMany"]
  storageClassName: "nfs"
  resources:
    requests:
      storage: 100M

3)       修改Deployment资源配置

a．新增标签jmx=prometheus

b．新增监控启动项…

c．新增监控端口jmx

d．新增共享目录jmxprometheus

apiVersion: apps/v1
kind: Deployment
metadata:
  name: appservice
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: appservice
      
  template:
    metadata:
      labels:
        app: appservice
        jmx: prometheus
      annotations:
        version: "20200722"
    spec:
      containers:
      - name: appservice
        image: hub.com/test/appservice:latest
        imagePullPolicy: Always
        env:
        - name: JAVA_OPTS
          value: >
            -Dprops=/root/appservice.properties -Dlogpath=/root/logs
            -javaagent:jmxprometheus/AppMonitor.jar
            -javaagent:jmxprometheus/jmx_prometheus_javaagent-0.13.0.jar=8013:jmxprometheus/namenode.yaml
   -Dcom.sun.management.jmxremote
            -Djava.rmi.server.hostname=localhost
            -Dcom.sun.management.jmxremote.port=8012
            -Dcom.sun.management.jmxremote.rmi.port=8012
            -Dcom.sun.management.jmxremote.authenticate=false
            -Dcom.sun.management.jmxremote.ssl=false
            -Dcom.sun.management.jmxremote.access.file=jmx
            -Dcom.sun.management.jmxremote.password.file=jmx
            -noverify
        ports:
        - name: dubbo
          containerPort: 20005
        - name: rest
          containerPort: 8113
        - name: jmx
          containerPort: 8013
        livenessProbe:
          initialDelaySeconds: 60
          timeoutSeconds: 10
          tcpSocket:
            port: 20005
        resources:
          limits:
            memory: 1Gi
        volumeMounts:
        - name: config
          mountPath: /root/appservice.properties
          subPath: appservice.properties- name: log
          mountPath: /root/logs
        - name: html
          mountPath: /root/html
        - name: jmxprometheus
          mountPath: /root/jmxprometheus
          
      volumes:
      - name: config
        configMap:
          name: appservice
      - name: log
        hostPath:
          path: /home/logs/appservice
          type: DirectoryOrCreate
      - name: jmxprometheus
        persistentVolumeClaim:
          claimName: jmxprometheus

4)       新增监控Service资源，用于动态发现应用PODIP

apiVersion: v1
kind: Service
metadata:
  name: jmxprometheus
  namespace: default
  labels:
    jmx: prometheus
spec:
  selector:
    jmx: prometheus
  ports: 
  - name: jmx
    port: 8013

5)       新增prometheus监控配置

- job_name: jmxprometheus
  honor_labels: false       # 将冲突标签重命名为exported_<original-label>
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - default
  scrape_interval: 30s      # 抓取间隔
  relabel_configs:
  - action: keep            # 收集匹配regex的源标签(service标签为jmx=prometheus)
    source_labels:
    - __meta_kubernetes_service_label_jmx
    regex: prometheus
  - action: keep            # 收集匹配regex的源标签(port的name=jmx)
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: jmx
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:      # 替换标签namespace、service、pod
    - __meta_kubernetes_namespace
    target_label: namespace    
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - target_label: endpoint
    replacement: jmx

三、监控展示：

注意：非集群机器无法直接访问PODIP，也就无法直接查看监控数据页面。