Prometheus中的黑盒监控blackbox

一、安装blackbox-exporter

地址：https://github.com/prometheus/blackbox_exporter

创建configmap

blackbox.yaml

modules:
  http_2xx:
    prober: http
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp

创建

# kubectl create cm blackbox-config --from-file=blackbox.yaml -n monitoring
configmap/blackbox-config created

创建blackcbox-exporter

blackbox-exporter.yaml

apiVersion: v1
kind: Service
metadata:
  name: blackbox
  namespace: monitoring
  labels:
    k8s-app: blackbox
spec:
  selector:
    k8s-app: blackbox
  ports:
  - name: blackbox
    port: 9115
    protocol: TCP
  type: ClusterIP

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blackbox
  namespace: monitoring
  labels:
    k8s-app: blackbox
spec:
  selector:
    matchLabels:
      k8s-app: blackbox
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: blackbox
    spec:
      containers:
      - name: blackbox
        image: prom/blackbox-exporter:master
        imagePullPolicy: IfNotPresent
        args: ["--config.file=/etc/blackbox/blackbox.yaml"]
        resources:
          requests:
            memory: 100M
        volumeMounts:
        - name: tz-config
          mountPath: /etc/localtime
          readOnly: true
        - name: blackbox-config
          mountPath: /etc/blackbox
          readOnly: true
      nodeSelector:
        monitor: "true"
      volumes:  
        - name: tz-config
          hostPath:
            path: /usr/share/zoneinfo/Asia/Shanghai
        - name: blackbox-config
          configMap:
            name: blackbox-config

创建

kubectl create -f blackbox-exporter.yaml

查看

# kubectl get svc,pod -n monitoring -l k8s-app=blackbox
NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/blackbox   ClusterIP   10.108.47.141   <none>        9115/TCP   10m

NAME                            READY   STATUS    RESTARTS   AGE
pod/blackbox-86b7486879-w6n22   1/1     Running   0          5m9s

测试

# curl "http://10.108.47.141:9115/probe?target=baidu.com&module=http_2xx"


# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 0.017669272
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 0.079658423
# HELP probe_failed_due_to_regex Indicates if probe failed due to regex
# TYPE probe_failed_due_to_regex gauge
probe_failed_due_to_regex 0
# HELP probe_http_content_length Length of http content response
# TYPE probe_http_content_length gauge
probe_http_content_length 81
# HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirects
# TYPE probe_http_duration_seconds gauge
probe_http_duration_seconds{phase="connect"} 0.031100739
probe_http_duration_seconds{phase="processing"} 0.030203142
probe_http_duration_seconds{phase="resolve"} 0.017669272
probe_http_duration_seconds{phase="tls"} 0
probe_http_duration_seconds{phase="transfer"} 0.0002315
# HELP probe_http_last_modified_timestamp_seconds Returns the Last-Modified HTTP response header in unixtime
# TYPE probe_http_last_modified_timestamp_seconds gauge
probe_http_last_modified_timestamp_seconds 1.26330408e+09
# HELP probe_http_redirects The number of redirects
# TYPE probe_http_redirects gauge
probe_http_redirects 0
# HELP probe_http_ssl Indicates if SSL was used for the final redirect
# TYPE probe_http_ssl gauge
probe_http_ssl 0
# HELP probe_http_status_code Response HTTP status code
# TYPE probe_http_status_code gauge
probe_http_status_code 200
# HELP probe_http_uncompressed_body_length Length of uncompressed response body
# TYPE probe_http_uncompressed_body_length gauge
probe_http_uncompressed_body_length 81
# HELP probe_http_version Returns the version of HTTP of the probe response
# TYPE probe_http_version gauge
probe_http_version 1.1
# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
# TYPE probe_ip_addr_hash gauge
probe_ip_addr_hash 2.752187856e+09
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 4
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 1

二、Blacakbox-exporter应用

 使用prometheus addition 功能，添加blackbox功能应用

地址：https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/additional-scrape-config.md　

先创建一个secret

prometheus-additional.yaml

- job_name: 'blackbox'
  metrics_path: /probe
  params:
    module: [http_2xx]  # Look for a HTTP 200 response.
  static_configs:
    - targets:
      #- http://prometheus.io    # Target to probe with http.
      #- https://prometheus.io   # Target to probe with https.
      - https://www.baidu.com # Target to probe with http on port 8080.
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox:9115  # The blackbox exporter's real hostname:port.

• 替换ip地址为blackbox的service名称

创建

kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run -oyaml > additional-scrape-configs.yaml

# kubectl create -f additional-scrape-configs.yaml -n monitoring
secret/additional-scrape-configs created

查看

# kubectl get secret -n monitoring
NAME                              TYPE                                  DATA   AGE
additional-scrape-configs         Opaque    

修改Prometheus的CRD

$path/prometheus/kube-prometheus/manifests/prometheus-prometheus.yaml

  serviceAccountName: prometheus-k8s
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional.yaml
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}

• 新增additionalScrapeConfigs配置项

# kubectl replace -f prometheus-prometheus.yaml

　　重启Prometheus，不能自动启动，就先删除Prometheus的pod

查看prometheus中status/configuration

导入grafana的dashboard，可以在grafana官网上寻找