Prometheus 使用 blackbox 监控 http tcp

Blackbox_exporter

blackbox_exporter 是 Prometheus 拿来对 http/https、tcp、icmp、dns、进行的黑盒监控工具
什么是黑盒监控？以下介绍是抄 zhangguanzhang 的 prometheus的黑盒监控

常规的各种exporter都是和需要监控的机器一起安装的,如果需要监控一些tcp端口和七层应用层的状态呢,这个时候就需要黑盒监控了,不需要安装在目标机器上即可从外部去监控。

---

安装

二进制安装

wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.linux-amd64.tar.gz
tar zxvf blackbox_exporter-0.16.0.linux-amd64.tar.gz
cd blackbox_exporter-0.16.0.linux-amd64
./blackbox_exporter <flags>

blackbox_exporter 有用的参数大概是如下几个

# ./blackbox_exporter --help
usage: blackbox_exporter [<flags>]

Flags:
  -h, --help                     Show context-sensitive help (also try --help-long and --help-man).
      --config.file="blackbox.yml"  
                                 Blackbox exporter configuration file.
      --web.listen-address=":9115"  
                                 The address to listen on for HTTP requests.

      --log.level=info           Only log messages with the given severity or above. One of: [debug, info, warn, error]

启动

# 默认端口为9115
nohup ./blackbox_exporter --config.file="blackbox.yml" &

docker安装

由于 Prometheus 所有组件默认时区都使用的 UTC ，所以推荐用 Docker 去运行它

# 如果你不需要开 debug，请去掉最后的 --log.level=debug
docker run --rm -d -p 9115:9115 --name blackbox_exporter -v /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime:ro -v /data/prometheus/blackbox_exporter/blackbox.yml:/config/blackbox.yml prom/blackbox-exporter:master --config.file=/config/blackbox.yml --log.level=debug

blackbox.yml

# 官方默认的配置文件
modules:
  http_2xx:
    prober: http
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp

配置

prometheus.yml

HTTP 配置

scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  # 模块对应 blackbox.yml 
    static_configs:
      - targets:
        - http://baidu.com    # http
        - https://baidu.com   # https
        - http://xx.com:8080 # 8080端口的域名
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115  # blackbox安装在哪台机器

TCP 配置

  - job_name: blackbox_tcp
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
        - 192.168.1.2:280
        - 192.168.1.2:7013

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.1.99:9115 # Blackbox exporter.

重启 Prometheus ，打开 targets ，即可看到。

想查看有哪些指标，则点击 Endpoint 下面的 URL 即可：

HTTP 比较值得关注的有如下几项：

# 返回的 http 状态码
probe_http_status_code 200
# https 证书过期时间，默认使用 unixtime
probe_ssl_earliest_cert_expiry 1.637745419e+09
# 如果探测成功则为 1，反之为 0
probe_success 1

告警规则

# 以下两条二选一
groups:
  - name: http
    rules:
    - alert: xxx域名解析失败
      expr: probe_success{instance="https://xx.com"} == 0
      for: 1m
      labels:
        severity: "error"
      annotations:
        summary: "xxx域名解析失败"
    - alert: xxx域名解析失败
      expr: probe_http_status_code{instance="https://xx.com"} != 200
      for: 5m
      labels:
        severity: "error"
      annotations:
        summary: "xxx域名解析失败"

自定义模块

有时可能对于某些 URL 需要带参数，如 header、body 之类的，就需要自定义一个模块，官方例子。

编辑 blackbox.yml

  http_2xx_wxjj:
    prober: http
    timeout: 5s
    http:
      method: GET
      headers:
        Cookie: JSESSIONID=C123455dfdf
        sid: 41c912344555-24rkjkffd
        appid: 1221kj2h1k3hjk13hk
      body: '{}'

编辑 Prometheus.yml

  - job_name: 'blackbox_wxjl'
    metrics_path: /probe
    params:
      module: [http_2xx_wxjj]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - http://192.168.201.173:808/byxxxxx/41234456661f-4357c9?head=APP_GeList&user=%E9%BB%84%E5%AE%15
   # Target to probe with http.

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.18.11.154:9115  # The blackbox exporter's real hostname:port.

开启 debug

当你觉得自己设置没错，http 状态码却返回不正确，想要调试一下，就需要打开debug。

• 启动时指定 --log.level=debug
• targets 后面带上 &debug=true，即 http://172.18.11.154:9115/probe?module=http_2xx_wxjj&target=http://192.168.201.173:808/byxxxxx/41234456661f-4357c9?head=APP_GeList&user=黄��&debug=true

targets 开启 debug 会比正常链接输出更多信息

Module configuration:
prober: http
timeout: 5s
http:
    ip_protocol_fallback: true
    method: GET
    headers:
        Cookie: JSESSIONID=C123455dfdf
        appid: 41c912344555-24rkjkffd
        sid: 1221kj2h1k3hjk13hk
    body: '{}'
tcp:
    ip_protocol_fallback: true
icmp:
    ip_protocol_fallback: true
dns:
    ip_protocol_fallback: true

FAQ

需要安装多少个 blackbox_exporter ？
理论上只安装一个即可，在特别的网络环境，比如政务云，和第三方对接的时候，只有几台机器开通了网络，那就需要在那几台机器中的其中一台部署 blackbox_exporter，同时 Prometheus.yml 里的 replacement 填上相应的 ip