Prometheus监控学习笔记之使用JMX Exporter监控微服务JVM

0x00 概述

本文实现微服务JVM监控的方法为，使用volume HostPath挂载的JMX Exporter的方式在容器内以in-process的方式实现对微服务的JMV监控。

主要坑点为HostPath挂载JMX Exporter javaagent导致微服务无法启动。

0x02 部署

2.1 下载JMX Exporter javaagent包

$ mkdir -p /tmp/jmx_exporter
$ wget -O /tmp/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.17.0/jmx_prometheus_javaagent-0.17.0.jar

2.2 配置YAML文件

此处镜像采用Deployment+Tomcat的镜像作为样例，自测的时候注意及时更换镜像；

'''
1. 采用HostPath的方式，将下载的JMX Exporter的jar包挂载到服务器容内启动；
2. JMX Exporter的配置文件以configmap的形式挂载到服务容器内；
'''

Tomcat服务的Deployment文件：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-with-jvmExporter
  namespace: default
  labels:
    app: tomcat-with-jvmExporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tomcat-with-jvmExporter
  template:
    metadata:
      labels:
        app: tomcat-with-jvmExporter
    spec:
      nodeSelector:
        kubernetes.io/hostname: docker49
      restartPolicy: Always
      containers:
      - name: tomcat-with-jvmExporter
        image: tomcat:9.0.54-jdk8-openjdk
        imagePullPolicy: Always
        env:
        # JVM env
        - name: JAVA_OPTS
          value: "-Xms500m -Xmx500m -javaagent:/jmx_prometheus_javaagent-0.17.0.jar=5555:/jmx/prometheus-jmx-config.yaml"
          #value: "-Xms500m -Xmx500m"
        volumeMounts:
        - mountPath: /jmx_prometheus_javaagent-0.17.0.jar
          name: jmx-prometheus-javaagent
        - mountPath: /jmx
          name: prometheus-jmx-config
        resources:
          limits:
            cpu: 2000m
            memory: "4096Mi"
          requests:
            cpu: 2000m
            memory: "4096Mi"
      volumes:
      - name: jmx-prometheus-javaagent
        hostPath:
          path: /tmp/jmx_prometheus_javaagent-0.17.0.jar
      - configMap:
          name: prometheus-jmx-config
        name: prometheus-jmx-config

'''
1. 通过hostpath挂载的方式，对于pod来说，不一定会调度到这个节点服务器上；

2. 如果pod没调度到有这个jar包的服务器上，在启动pod的时候，会发现pod挂载了jar包和配置文件，但是无法执行这个jar包；

3. 所以需要在yaml中通过nodeSelector指定pod调度到固定节点上，这样hostPath才能取到文件，pod才会成功挂载这个文件并正常运行

4. 上面我们将jar包下载到了docker49这台机器上，所以指定将tomcat的pod调度到docker49，注意红字部分

'''

未配置nodeSelector的话，会出现如下报错：

'''
Error opening zip file or JAR manifest missing : /jmx_prometheus_javaagent-0.17.0.jar
Error occurred during initialization of VM
agent library failed to init: instrument
'''

直接搜索以上报错日志，StackOverFlow上大部分都会说是权限问题，但是无论你改成755还是777都一样，配置nodeSelector，将pod调度到JMX Exporter jar包的机器上即可。

Tomcat服务的Service文件：

apiVersion: v1
kind: Service
metadata:
  name: tomcat-with-jvmExporter-svc
  namespace: default
  labels:
    app: tomcat-with-jvmExporter-svc
  annotations:
    prometheus.io/port: "5555"
    prometheus.io/jvm: "true"
spec:
  type: NodePort
  ports:
  - port: 8080
    name: tomcat-with-jvmExporter-svc
    targetPort: 30899
    nodePort: 30899
  - name: jmx-metrics
    port: 5555
    protocol: TCP
  selector:
    app: tomcat-with-jvmExporter

Tomcat服务的configmap文件（此处为方便测试采用最小化配置文件，官方配置文件样例）：

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-jmx-config
  namespace: default
data:
  prometheus-jmx-config.yaml: |
#    lowercaseOutputLabelNames: true
#    lowercaseOutputName: true
    rules:
    - pattern: ".*"
#        - pattern: 'Catalina<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+):'
#          name: tomcat_$3_total
#          labels:
#            port: "$2"
#            protocol: "$1"
#          help: Tomcat global $3
#          type: COUNTER
#        - pattern: 'Catalina<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|maxTime|processingTime|errorCount):'
#          name: tomcat_servlet_$3_total
#          labels:
#            module: "$1"
#            servlet: "$2"
#          help: Tomcat servlet $3 total
#          type: COUNTER
#        - pattern: 'Catalina<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount):'
#          name: tomcat_threadpool_$3
#          labels:
#            port: "$2"
#            protocol: "$1"
#          help: Tomcat threadpool $3
#          type: GAUGE
#        - pattern: 'Catalina<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
#          name: tomcat_session_$3_total
#          labels:
#            context: "$2"
#            host: "$1"
#          help: Tomcat session $3 total
#          type: COUNTER

2.3 All in One yaml文件

tomcat-with-jvmExporter-AllinOne.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-wtih-jvmExporter
  namespace: default
  labels:
    app: tomcat-with-jvmExporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tomcat-with-jvmExporter
  template:
    metadata:
      labels:
        app: tomcat-with-jvmExporter
    spec:
      nodeSelector:
        kubernetes.io/hostname: docker49
      restartPolicy: Always
      containers:
      - name: tomcat-with-jvmExporter
        image: tomcat:9.0.54-jdk8-openjdk
        imagePullPolicy: Always
        env:
        # JVM env
        - name: JAVA_OPTS
          value: "-Xms500m -Xmx500m -javaagent:/jmx_prometheus_javaagent-0.17.0.jar=5555:/jmx/prometheus-jmx-config.yaml"
          #value: "-Xms500m -Xmx500m"
        volumeMounts:
        - mountPath: /jmx_prometheus_javaagent-0.17.0.jar
          name: jmx-prometheus-javaagent
        - mountPath: /jmx
          name: prometheus-jmx-config
        resources:
          limits:
            cpu: 2000m
            memory: "4096Mi"
          requests:
            cpu: 2000m
            memory: "4096Mi"
      volumes:
      - name: jmx-prometheus-javaagent
        hostPath:
          path: /tmp/jmx_prometheus_javaagent-0.17.0.jar
      - configMap:
          name: prometheus-jmx-config
        name: prometheus-jmx-config
---
apiVersion: v1
kind: Service
metadata:
  name: tomcat-with-jvmExporter-svc
  namespace: default
  labels:
    app: tomcat-with-jvmExporter-svc
  annotations:
    prometheus.io/port: "5555"
    prometheus.io/jvm: "true"
spec:
  type: NodePort
  ports:
  - port: 8080
    name: tomcat-with-jvmExporter-svc
    targetPort: 30899
    nodePort: 30899
  - name: jmx-metrics
    port: 5555
    protocol: TCP
  selector:
    app: tomcat-with-jvmExporter
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-jmx-config
  namespace: default
data:
  prometheus-jmx-config.yaml: |
    #lowercaseOutputLabelNames: true
    #lowercaseOutputName: true
    rules:
    - pattern: ".*"
#        - pattern: 'Catalina<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+):'
#          name: tomcat_$3_total
#          labels:
#            port: "$2"
#            protocol: "$1"
#          help: Tomcat global $3
#          type: COUNTER
#        - pattern: 'Catalina<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|maxTime|processingTime|errorCount):'
#          name: tomcat_servlet_$3_total
#          labels:
#            module: "$1"
#            servlet: "$2"
#          help: Tomcat servlet $3 total
#          type: COUNTER
#        - pattern: 'Catalina<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount):'
#          name: tomcat_threadpool_$3
#          labels:
#            port: "$2"
#            protocol: "$1"
#          help: Tomcat threadpool $3
#          type: GAUGE
#        - pattern: 'Catalina<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
#          name: tomcat_session_$3_total
#          labels:
#            context: "$2"
#            host: "$1"
#          help: Tomcat session $3 total
#          type: COUNTER

2.4 启动并检查服务

启动服务：

kubectl apply -f tomcat-with-jvmExporter-AllinOne.yaml

 检查服务：

kubectl get svc -n default

 检查JVM Exporter是否已获取JVM监控数据，此处docker49 ip：192.168.6.6，JVM Exporter对外nodeExporter为30566

curl 192.168.6.6:30566/metrics

 如果返回以下数据，说明已成功获取监控数据：

jvm_threads_daemon 36.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 80.0
# HELP jvm_threads_started_total Started thread count of a JVM
# TYPE jvm_threads_started_total counter
jvm_threads_started_total 155322.0
# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
# TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0
# HELP jvm_threads_state Current count of threads by state
# TYPE jvm_threads_state gauge
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="TIMED_WAITING",} 22.0
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="RUNNABLE",} 12.0
jvm_threads_state{state="WAITING",} 41.0
# HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_currently_loaded gauge
jvm_classes_currently_loaded 18332.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 18478.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 146.0
# HELP jmx_config_reload_failure_created Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_created gauge
jmx_config_reload_failure_created 1.655778604558E9
# HELP jmx_config_reload_success_created Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_created gauge
jmx_config_reload_success_created 1.655778604539E9
# HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_created gauge
jvm_memory_pool_allocated_bytes_created{pool="Code Cache",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="PS Eden Space",} 1.655778606951E9
jvm_memory_pool_allocated_bytes_created{pool="PS Old Gen",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="PS Survivor Space",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.655778606954E9

0x03 配置Prometheus获取监控个数据

修改prometheus.yml配置文件，新增JVM Exporter数据

  - job_name: "jvm-ucm-config"
      honor_labels: true
      static_configs:
        - targets: ["192.168.6.6:30566"]

 重新加载Prometheus配置文件

kill -HUP $pid

0x04 Grafana导入JVM监控面板

推荐面板id：11278

需要修改面板变量，根据以上配置，变量设置为2，job+instance

0x05 参考

prometheus/jmx_exporter官方Github

K8S 中使用 Prometheus 监控 JVM (一)