Filebeat + Logstash + MongoDB

Filebeat + Logstash + MongoDB

简介

​ Logstash是一个开源数据收集引擎，具有实时管道功能。Logstash可以动态地将来自不同数据源的数据统一起来，并将数据标准化到你所选择的目的地。

jdk安装

• 下载镜像

  wget https://mirrors.huaweicloud.com/java/jdk/8u151-b12/jdk-8u151-linux-x64.tar.gz
  

• 解压

  tar zxvf jdk-8u151-linux-x64.tar.gz -C /usr/local/
  

• 改名

  cd ../
  mv jdk1.8.0_151 jdk1.8
  

• 注册到环境变量中(/etc/profile)

  export JAVA_HOME=/usr/local/jdk1.8
  export JRE_HOME=${JAVA_HOME}/jre
  export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
  export PATH=.:${JAVA_HOME}/bin:$PATH
  

• source 执行

  source /etc/profile
  

• 检查版本

  java -version
  

filebeat

• 下载

  wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.15.0-linux-x86_64.tar.gz
  

• 解压

  tar zxvf filebeat-7.15.0-linux-x86_64.tar.gz -C /usr/local/
  

• 改名

  cd ../
  mv filebeat-7.15.0-linux-x86_64/ filebeat
  

• 配置

  filebeat.inputs:
  - type: log
    enabled: false
    paths:
      - /var/log/*.log
      
  output.logstash:
    hosts: ["localhost:5044"]
    # Optional SSL. By default is off.
    # List of root certificates for HTTPS server verifications
    #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
    # Certificate for SSL client authentication
    #ssl.certificate: "/etc/pki/client/cert.pem"
    # Client Certificate Key
    #ssl.key: "/etc/pki/client/cert.key"
  

• 启动

  # 前端启动
  ./filebeat -e -c filebeat.yml
  # 后台启动
  nohup ./filebeat -e -c filebeat.yml &
  

logstash安装

• 下载

  wget https://artifacts.elastic.co/downloads/logstash/logstash-7.15.0-linux-x86_64.tar.gz
  

• 解压

  tar zxvf logstash-7.15.0-linux-x86_64.tar.gz -C /usr/local/
  

• 改名

  cd ../
  mv logstash-7.15.0/ logstash
  

• 简单的测试

  bin/logstash -e 'input { stdin {} } output { stdout {} }'
  

  

• 修改配置( /usr/local/logstash/config/application/operator_mongo.conf )

  # 输入
  input {
      beats {
          port => "5044"
      }
  }
  
  # 过滤
  filter {
      grok {
          match => {"message" => "%{COMBINEDAPACHELOG}"}
      }
      geoip {
          source => "clientip"
      }
  }
  
  # 输出
  output {
      mongodb {
          codec => line {format => "%{message}"}
          uri => "mongodb://localhost:27017/admin"
          database => "test"
          collection => "trace_log"
      }
      stdout { codec => rubydebug }
  }
  

• 检查配置

  bin/logstash -f /usr/local/logstash/config/application/operator_mongo.conf --config.test_and_exit
  

  

• 单logstash启动(指定文件名称)

  bin/logstash -f config/application/operator_mongo.conf --config.reload.automatic
  

  

• 多logstash启动(精确到文件夹就可以了)

  bin/logstash -f config/application --config.reload.automatic