使用Prometheus和Grafana监控服务

Prometheus是一个开源的监控服务，可以用来采集服务的状态数据，Grafana是一个开源的分析和监控软件，我最近在配置这些服务来对我们现有的服务以及机器进行监控。这篇文章记录了配置Prometheus和Grafana的步骤，以便以后查阅使用，本文使用的服务器是CentOS 7。

安装Prometheus

创建本地用户

sudo useradd prometheus
sudo mkdir /etc/prometheus
sudo chown prometheus:prometheus /etc/prometheus

下载Prometheus

# 下载地址https://prometheus.io/download/
# 切换到Prometheus用户
sudo su - prometheus
# 下载prometheus安装文件
wget https://github.com/prometheus/prometheus/releases/download/v2.6.0/prometheus-2.6.0.linux-amd64.tar.gz
# 解压文件
tar xzf prometheus-2.0.0.linux-amd64.tar.gz
# 将可执行文件复制到/usr/local/bin目录下
sudo cp prometheus-2.6.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.6.0.linux-amd64/promtool /usr/local/bin/
# 将Prometheus的界面模版复制到/etc/prometheus目录下
cp -R prometheus-2.6.0.linux-amd64/consoles /etc/prometheus
cp -R prometheus-2.6.0.linux-amd64/console_libraries /etc/prometheus

配置Prometheus

配置文件路径/etc/prometheus/prometheus.yml

文件内容如下：

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
 
  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'
 
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
 
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
 
    static_configs:
      - targets: ['localhost:9090']

　　

配置Prometheus为系统服务

添加服务文件/etc/systemd/system/prometheus.servie

文件内容如下：

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus 
    --config.file /etc/prometheus/prometheus.yml 
    --storage.tsdb.path /home/prometheus/data/prometheus/ 
    --web.console.templates=/etc/prometheus/consoles 
    --web.console.libraries=/etc/prometheus/console_libraries 
    --web.external-url=http://<monitor.domain.name>
[Install]
WantedBy=multi-user.target

启动Prometheus服务

# 启动服务
sudo systemctl start prometheus
# 设置服务开机时自动重启
sudo systemctl enable prometheus
# 查看服务配置状态
sudo systemctl status prometheus

配置nginx使用域名访问

upstream http_monitor {
    server localhost:9090;
    keepalive 300;
}
server {
    listen 80;
    server_name <prometheus.domain.name>;
    location / {
        auth_basic "Prometheus server authentication";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://http_monitor;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffers 4096 32k;
        proxy_buffer_size  32K;
        proxy_busy_buffers_size 32k;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_cache_bypass $http_upgrade;
    }
}

 

上面的 auth_basic_user_file /etc/nginx/.htpasswd; 配置了服务访问时需要输入密码验证，该文件使用htpasswd 生成，该工具在httpd-tools中。

sudo yum install httpd-tools
sudo htpasswd -c /etc/nginx/.htpasswd user1

安装grafana

# 介绍页面https://grafana.com/grafana/download?platform=linux
wget https://dl.grafana.com/oss/release/grafana-5.4.2-1.x86_64.rpm 
sudo yum localinstall grafana-5.4.2-1.x86_64.rpm

 

Grafana配置文件

# Grafana的配置文件路径：/etc/grafana/grafana.ini

Grafana的数据文件路径

# Grafana数据文件路径：/var/lib/grafana

配置nginx使用域名访问

配置文件: /etc/nginx/conf.d/grafana.conf

 

upstream http_grafana_monitor {
    server localhost:3000;
    keepalive 300;
}
server {
    listen 80;
    server_name <grafana.domain.name>;
    location / {
        proxy_pass http://http_grafana_monitor;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffers 4096 32k;
        proxy_buffer_size  32K;
        proxy_busy_buffers_size 32k;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_cache_bypass $http_upgrade;
    }
}