• elk实战分析nginx日志文档


    elk实战分析nginx日志文档

    架构:
      kibana <--- es-cluster <--- logstash <--- filebeat

    环境准备:
    192.168.3.1 node1 node1.xkops.com 内存2G
    192.168.3.2 node2 node2.xkops.com
    192.168.3.3 node3 node3.xkops.com

    ---------------elasticserach安装部分----------------
    1.在node1|node2上安装jdk,安装elasticsearch并配置集群。
    node1:
    ①.安装jdk1.8_65
    [root@node1 ~]# rpm -ivh jdk-8u65-linux-x64.rpm
    ②.下载安装elasticsearch,并设置开启自启动。
    [root@node1 ~]# wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.4/elasticsearch-2.3.4.rpm
    [root@node1 ~]# rpm -ivh elasticsearch-2.3.4.rpm
    [root@node1 ~]# chkconfig --add elasticsearch
    ③.修改配置文件,并启动服务。
    [root@node1 ~]# grep -Ev "^#|^$" /etc/elasticsearch/elasticsearch.yml
    cluster.name: elk-xkops
    node.name: node-1
    network.host: 192.168.3.1
    http.port: 9200
    discovery.zen.ping.unicast.hosts: ["node2"]
    [root@node1 ~]# service elasticsearch start
    node2:
    ①.安装jdk1.8_65
    [root@node2 ~]# rpm -ivh jdk-8u65-linux-x64.rpm
    ②.下载安装elasticsearch,并设置开启自启动。
    [root@node2 ~]# wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.4/elasticsearch-2.3.4.rpm
    [root@node2 ~]# rpm -ivh elasticsearch-2.3.4.rpm
    [root@node2 ~]# chkconfig --add elasticsearch
    ③.修改配置文件,并启动服务。
    [root@node2 ~]# grep -Ev "^#|^$" /etc/elasticsearch/elasticsearch.yml
    cluster.name: elk-xkops
    node.name: node-1
    network.host: 192.168.3.2
    discovery.zen.ping.unicast.hosts: ["node1"]
    [root@node2 ~]# service elasticsearch start
    *此时集群配置完成。

    2.elasticsearch常用插件安装:
    只需在node1(master)上安装即可:
    在线安装:
    ①.安装head插件。
    [root@node1 ~]# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
    ②.安装bigdesk插件。
    [root@node1 ~]# /usr/share/elasticsearch/bin/plugin install hlstudio/bigdesk
    ③.安装kopf插件。
    [root@node1 ~]# /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf
    离线安装:
    ①.安装head插件。
    [root@node1 ~]# /usr/share/elasticsearch/bin/plugin install file:/elk/soft/elasticsearch-head-master.zip
    ②.安装bigdesk插件。
    [root@node1 ~]# /usr/share/elasticsearch/bin/plugin install file:/elk/soft/bigdesk-master.zip
    ③.安装kopf插件。
    [root@node1 ~]# /usr/share/elasticsearch/bin/plugin install file:/elk/soft/elasticsearch-kopf-master.zip
    浏览器端访问插件:
    http://192.168.3.1:9200/_plugin/head
    http://192.168.3.1:9200/_plugin/bigdesk
    http://192.168.3.1:9200/_plugin/kopf


    ---------------logstash安装部分----------------
    1.在node1上安装logstash。(需要jdk,已经安装)
    [root@node1 ~]# wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.3.4-1.noarch.rpm
    [root@node1 ~]# rpm -ivh logstash-2.3.4-1.noarch.rpm

    2.编辑配置文件
    [root@node1 ~]# cat /etc/logstash/conf.d/logstash.conf

    input {
        beats {
            port => 5044
            codec => "json"
        }
    }
    filter{
        if [type] == "nginx"{
        date{
            locale => "en"
            match => ["@timestamp", "UNIX_MS"]
            target => "@timestamp"
        }}
    }
    output{
        if [type] == "nginx"{
        elasticsearch {
            hosts => ["192.168.3.1:9200"]
            index => "nginx-%{+YYYY.MM.dd}"
            flush_size => 2000
            idle_flush_time => 10
        }}
    }

    3.启动logstash服务
    [root@node1 ~]# service logstash start

    ---------------kibana安装部分----------------
    1.在node1上安装kibana
    [root@node1 ~]# wget https://download.elastic.co/kibana/kibana/kibana-4.5.2-1.x86_64.rpm
    [root@node1 ~]# rpm -ivh kibana-4.5.2-1.x86_64.rpm

    2.配置kibana连接elasticsearch。
    [root@node1 ~]# grep -Ev "^#|^$" /opt/kibana/config/kibana.yml
    server.port: 5601
    server.host: "192.168.3.1"
    elasticsearch.url: "http://192.168.3.1:9200"
    elasticsearch.preserveHost: true
    kibana.index: ".kibana"
    kibana.defaultAppId: "discover"
    elasticsearch.requestTimeout: 30000
    elasticsearch.shardTimeout: 0
    elasticsearch.startupTimeout: 5000

    3.启动kibana服务
    [root@node1 ~]# service kibana start


    ----------------nginx安装部分-----------------
    1.安装nginx和httpd-tools工具包。
    [root@node1 ~]# yum -y install nginx httpd-tools
    2.添加访问kibana的用户,并设置密码。
    [root@node1 ~]# htpasswd -c /etc/nginx/htpasswd.users kibanaadmin
    3.反向代理kibana服务,配置文件如下:
    [root@node1 ~]# cat /etc/nginx/conf.d/kibana.conf

    server {
        listen 80;
    
        server_name elk.xkops.com;
    
        auth_basic "Restricted Access";
        auth_basic_user_file /etc/nginx/htpasswd.users;
    
        location / {
            proxy_pass http://192.168.3.1:5601;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;        
        }
    }

    4.浏览器访问
    http://elk.xkops.com


    ----------------filebeat安装部分-----------------
    1.在node3上安装filebeat。
    下载官方示例数据(nginx日志):
    [root@node3 ~]# wget https://download.elastic.co/demos/kibana/gettingstarted/logs.jsonl.gz

    下载软件包:
    [root@node3 ~]# wget https://download.elastic.co/beats/filebeat/filebeat-1.2.3-x86_64.rpm
    [root@node3 ~]# rpm -ivh /elk/filebeat-1.2.3-x86_64.rpm

    2.编辑filebeat配置文件。
    *提示:修改logs.jsonl文件内容,取其中偶数行。(sed)
    sed -n 'n;p' logs.jsonl >> nginx.log 或者sed -n '2~2p' logs.jsonl >> nginx.log

    [root@node3 ~]# grep -Ev "#|^$" /etc/filebeat/filebeat.yml

    filebeat:
      prospectors:
        -
          paths:
            - /root/nginx.json
          document_type: nginx
    
      registry_file: /var/lib/filebeat/registry
    output:
      logstash:
        hosts: ["192.168.3.1:5044"]
    shipper:
    logging:
      files:

    3.创建mapping映射。
    [root@node3 ~]# curl -XPOST 'http://192.168.3.1:9200/_template/filebeat?pretty' -d@/etc/filebeat/filebeat.template.json

    4.启动filebeat服务。
    [root@node3 ~]# service filebeat start

    至此,整个elk搭建完成,可以在kibana展示端进行各种操作,比如检索日志,制作各种展示图表了。

  • 相关阅读:
    网站迁移服务器后CPU、内存飙升,设置robots.txt 问题
    System.Web.Mvc 找到的程序集清单定义与程序集引用不匹配
    滑动窗口协议
    TCP拥塞控制
    计网常用协议
    TCP协议中的三次握手、四次挥手
    浏览网页的详细过程
    docker网络模式
    openstack网络
    查找算法
  • 原文地址:https://www.cnblogs.com/xkops/p/5761191.html
Copyright © 2020-2023  润新知