部署ELK收集Nginx日志（redis缓存 Filebeat收集）

Filebeat连接Redis数据库：

当前已经把 Nginx日志配置为 json格式了（要修改nginx日志的格式，请复制下方日志配置到nginx配置中）
如果没有安装nginx，请点击下方文章链接

CentOS-7.5，源码编译Nginx-1.14.2详解

Nginx (engine x) 是一个高性能的HTTP和反向代理服务，也是一个IMAP/POP3/SMTP服务。Nginx是由伊戈尔·赛索耶夫为俄罗斯访问量第二的Rambler.ru站点（俄文：Ра...

1526

log_format main_json '{"client_ip": "$remote_addr",'
'"client_user": "$remote_user",'
'"local_time": "[$time_local]",'
'"request": "$request",'
'"response_time": "$request_time",'
'"upstream_time": "$upstream_response_time",'
'"status_num": "$status",'
'"response_size": "$body_bytes_sent",'
'"skip_link": "$http_referer",'
'"client_agent": "$http_user_agent",'
'"new_subject": "$http_x_forwarded_for"'
'}';

access_log  /var/log/nginx/access.log  main_json;

务必要关闭模板功能，都设置为 nginx.yml.disabled 关闭状态
```
[root@web01 ~]# filebeat modules disable nginx
Disabled nginx
```

安装redis数据库：

[root@web01 ~]# yum -y install redis
[root@web01 ~]# vim /etc/redis.conf +61
bind 0.0.0.0

[root@web01 ~]# systemctl start redis.service
[root@web01 ~]# systemctl enable redis.service
[root@web01 ~]# netstat -tunpl |grep redis
tcp        0      0 0.0.0.0:6379          0.0.0.0:*               LISTEN      2181/redis-server 1

安装filebeat包：

[root@web01 ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.6.0-x86_64.rpm
[root@web01 ~]# rpm -ivh filebeat-6.6.0-x86_64.rpm
warning: filebeat-6.6.0-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:filebeat-6.6.0-1                 ################################# [100%]

配置filebeat组件：

[root@web01 ~]# vim /etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  enabled: true    # 启用日志
  paths:
    - /var/log/nginx/access.log   # 指定nginx日志
  json.keys_under_root: true
  json.overwrite_keys: true
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false    # 关闭模板
outout.redis:
    hosts: ["172.18.1.100"]    # redis地址
    key: "filebeat"    # 在reidis内创建的key名称
    db: 0
    timeout: 5

重启filebeat组件：

[root@web01 ~]# systemctl restart filebeat.service

查看redis数据数量：

当前已经清空redis所有数据，也清空了nginx日志文件

# llen 查看filebeat键的值有多少数量
[root@web01 ~]# redis-cli llen filebeat
(integer) 0

# ab 压力测试 1000个请求
[root@web01 ~]# ab -n 1000 -c 1000 http://172.18.1.100/

# 此时日志已经缓存到了redis数据库中
[root@web01 ~]# redis-cli llen filebeat
(integer) 1000

[root@web01 ~]# redis-cli rpop filebeat
"{"@timestamp":"2019-12-17T06:45:43.875Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.6.0"},"skip_link":"-","log":{"file":{"path":"/var/log/nginx/access.log"}},"local_time":"[17/Dec/2019:14:45:43 +0800]","input":{"type":"log"},"host":{"name":"web01.novalocal"},"source":"/var/log/nginx/access.log","offset":233766,"new_subject":"-","beat":{"hostname":"web01.novalocal","version":"6.6.0","name":"web01.novalocal"},"client_ip":"172.18.1.100","client_agent":"ApacheBench/2.3","client_user":"-","request":"GET / HTTP/1.0","status_num":"200","response_size":"11","prospector":{"type":"log"}}"

配置Logstash拉取redis数据日志：

[root@logstash ~]# yum install java-1.8.0-openjdk -y
[root@logstash ~]# java -version

openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

下载并安装软件包：

[root@logstash ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.6.0.rpm
[root@logstash ~]# rpm -ivh logstash-6.6.0.rpm 
warning: logstash-6.6.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:6.6.0-1               ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Successfully created system startup script for Logstash

配置logstash抓取redis中nginx日志：

[root@logstash ~]# vim /etc/logstash/conf.d/nginx.conf

input {
  redis {
    host => "172.18.1.100"    # redis地址
    port => "6379"    # redis端口
    db => "0"
    key => "filebeat"   # redis中filebeat组件的key名称
    data_type => "list"
  }
}

filter {
  mutate {
    convert => ["response_time","float"]    # 
    convert => ["upstream_time","float"]    # 
  }
}

output {
  elasticsearch {
    hosts => "http://172.18.1.76:9200"     # elasticsearch主机地址
    manage_template => false
    index => "nginx-%{+YYYY.MM}"       # 输入到elasticsearch后的索引名称
  }
}

查看redis数据数量：

当 logstash拉取redis数据后会清理已经拉取的数据日志，最后是redis数据为空。

[root@web01 ~]# redis-cli llen filebeat
(integer) 1000

启动logstash组件：

[root@logstash ~]# systemctl start logstash
[root@logstash ~]# systemctl enable logstash

再次查看redis数据数量：

[root@web01 ~]# redis-cli llen filebeat
(integer) 0

查看elasticsearch中索引：

[root@db01 ~]# curl '172.18.1.76:9200/_cat/indices?v' 
health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   nginx-2019.12                   PhGgYHNHTm2vKiNepQuOFg   5   1      11000            0      3.1mb          1.5mb

安装Kibana页面：

[root@db01 ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-6.6.0-x86_64.rpm
[root@db01 ~]# rpm -ivh kibana-6.6.0-x86_64.rpm

配置Kibana连接Es：

[root@db01 ~]# vim /etc/kibana/kibana.yml

server.port: 5601                                # kibana监听的端口
server.host: "172.18.1.76"                       # kibana监听的IP地址
elasticsearch.hosts: ["http://172.18.1.76:9200"] # kibana连接ES集群master节点地址

启动Kibana：

[root@db01 ~]# systemctl start kibana.service
[root@db01 ~]# systemctl enable kibana.service
[root@db01 ~]# netstat -tunpl |grep 5601
tcp        0      0 172.18.1.76:5601        0.0.0.0:*               LISTEN      18542/node