elk实战分析nginx日志文档
架构:
kibana <--- es-cluster <--- logstash <--- filebeat
环境准备:
192.168.3.1 node1 node1.xkops.com 内存2G
192.168.3.2 node2 node2.xkops.com
192.168.3.3 node3 node3.xkops.com
---------------elasticserach安装部分----------------
1.在node1|node2上安装jdk,安装elasticsearch并配置集群。
node1:
①.安装jdk1.8_65
[root@node1 ~]# rpm -ivh jdk-8u65-linux-x64.rpm
②.下载安装elasticsearch,并设置开启自启动。
[root@node1 ~]# wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.4/elasticsearch-2.3.4.rpm
[root@node1 ~]# rpm -ivh elasticsearch-2.3.4.rpm
[root@node1 ~]# chkconfig --add elasticsearch
③.修改配置文件,并启动服务。
[root@node1 ~]# grep -Ev "^#|^$" /etc/elasticsearch/elasticsearch.yml
cluster.name: elk-xkops
node.name: node-1
network.host: 192.168.3.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["node2"]
[root@node1 ~]# service elasticsearch start
node2:
①.安装jdk1.8_65
[root@node2 ~]# rpm -ivh jdk-8u65-linux-x64.rpm
②.下载安装elasticsearch,并设置开启自启动。
[root@node2 ~]# wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.4/elasticsearch-2.3.4.rpm
[root@node2 ~]# rpm -ivh elasticsearch-2.3.4.rpm
[root@node2 ~]# chkconfig --add elasticsearch
③.修改配置文件,并启动服务。
[root@node2 ~]# grep -Ev "^#|^$" /etc/elasticsearch/elasticsearch.yml
cluster.name: elk-xkops
node.name: node-1
network.host: 192.168.3.2
discovery.zen.ping.unicast.hosts: ["node1"]
[root@node2 ~]# service elasticsearch start
*此时集群配置完成。
2.elasticsearch常用插件安装:
只需在node1(master)上安装即可:
在线安装:
①.安装head插件。
[root@node1 ~]# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
②.安装bigdesk插件。
[root@node1 ~]# /usr/share/elasticsearch/bin/plugin install hlstudio/bigdesk
③.安装kopf插件。
[root@node1 ~]# /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf
离线安装:
①.安装head插件。
[root@node1 ~]# /usr/share/elasticsearch/bin/plugin install file:/elk/soft/elasticsearch-head-master.zip
②.安装bigdesk插件。
[root@node1 ~]# /usr/share/elasticsearch/bin/plugin install file:/elk/soft/bigdesk-master.zip
③.安装kopf插件。
[root@node1 ~]# /usr/share/elasticsearch/bin/plugin install file:/elk/soft/elasticsearch-kopf-master.zip
浏览器端访问插件:
http://192.168.3.1:9200/_plugin/head
http://192.168.3.1:9200/_plugin/bigdesk
http://192.168.3.1:9200/_plugin/kopf
---------------logstash安装部分----------------
1.在node1上安装logstash。(需要jdk,已经安装)
[root@node1 ~]# wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.3.4-1.noarch.rpm
[root@node1 ~]# rpm -ivh logstash-2.3.4-1.noarch.rpm
2.编辑配置文件
[root@node1 ~]# cat /etc/logstash/conf.d/logstash.conf
input { beats { port => 5044 codec => "json" } } filter{ if [type] == "nginx"{ date{ locale => "en" match => ["@timestamp", "UNIX_MS"] target => "@timestamp" }} } output{ if [type] == "nginx"{ elasticsearch { hosts => ["192.168.3.1:9200"] index => "nginx-%{+YYYY.MM.dd}" flush_size => 2000 idle_flush_time => 10 }} }
3.启动logstash服务
[root@node1 ~]# service logstash start
---------------kibana安装部分----------------
1.在node1上安装kibana
[root@node1 ~]# wget https://download.elastic.co/kibana/kibana/kibana-4.5.2-1.x86_64.rpm
[root@node1 ~]# rpm -ivh kibana-4.5.2-1.x86_64.rpm
2.配置kibana连接elasticsearch。
[root@node1 ~]# grep -Ev "^#|^$" /opt/kibana/config/kibana.yml
server.port: 5601
server.host: "192.168.3.1"
elasticsearch.url: "http://192.168.3.1:9200"
elasticsearch.preserveHost: true
kibana.index: ".kibana"
kibana.defaultAppId: "discover"
elasticsearch.requestTimeout: 30000
elasticsearch.shardTimeout: 0
elasticsearch.startupTimeout: 5000
3.启动kibana服务
[root@node1 ~]# service kibana start
----------------nginx安装部分-----------------
1.安装nginx和httpd-tools工具包。
[root@node1 ~]# yum -y install nginx httpd-tools
2.添加访问kibana的用户,并设置密码。
[root@node1 ~]# htpasswd -c /etc/nginx/htpasswd.users kibanaadmin
3.反向代理kibana服务,配置文件如下:
[root@node1 ~]# cat /etc/nginx/conf.d/kibana.conf
server { listen 80; server_name elk.xkops.com; auth_basic "Restricted Access"; auth_basic_user_file /etc/nginx/htpasswd.users; location / { proxy_pass http://192.168.3.1:5601; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } }
4.浏览器访问
http://elk.xkops.com
----------------filebeat安装部分-----------------
1.在node3上安装filebeat。
下载官方示例数据(nginx日志):
[root@node3 ~]# wget https://download.elastic.co/demos/kibana/gettingstarted/logs.jsonl.gz
下载软件包:
[root@node3 ~]# wget https://download.elastic.co/beats/filebeat/filebeat-1.2.3-x86_64.rpm
[root@node3 ~]# rpm -ivh /elk/filebeat-1.2.3-x86_64.rpm
2.编辑filebeat配置文件。
*提示:修改logs.jsonl文件内容,取其中偶数行。(sed)
sed -n 'n;p' logs.jsonl >> nginx.log 或者sed -n '2~2p' logs.jsonl >> nginx.log
[root@node3 ~]# grep -Ev "#|^$" /etc/filebeat/filebeat.yml
filebeat: prospectors: - paths: - /root/nginx.json document_type: nginx registry_file: /var/lib/filebeat/registry output: logstash: hosts: ["192.168.3.1:5044"] shipper: logging: files:
3.创建mapping映射。
[root@node3 ~]# curl -XPOST 'http://192.168.3.1:9200/_template/filebeat?pretty' -d@/etc/filebeat/filebeat.template.json
4.启动filebeat服务。
[root@node3 ~]# service filebeat start
至此,整个elk搭建完成,可以在kibana展示端进行各种操作,比如检索日志,制作各种展示图表了。