filebeat系列快速调试demo

如何做热部署

热部署至少需要一个reload的yml文件，以及一个或多个具体处理逻辑的yml文件

reload.yml文件内容示例：输出到kafka

filebeat.config.inputs:
  enabled: true
  path: /opt/filebeat-7.6.2-linux-x86_64/prod-yml-test/*.yml
  reload.enabled: true
  reload.period: 3s
processors:
- rename:
   fields:
     - {from: "log.file.path",to: "log_file_path"}
     - {from: "agent.hostname",to: "log_host_name"}
- drop_fields:
   fields: ["log.file"]
output.kafka:
  hosts: ["10.192.78.27:9092"]
  topic: "sbdp_test"
http.enabled: true
http.host: 10.192.78.27
http.port: 5518

这里只要是   /opt/filebeat-7.6.2-linux-x86_64/prod-yml-test/*.yml   里面的yml文件有变化，会自动reload，频率是3s检查一次变化

接着配置一个kafka输入，放在上面定义的文件夹下面：

filebeat.inputs:
- type: kafka
  hosts: ["10.10.10.10:9092"]
  topics: ["aaa"]
  group_id: "group001"
  tags: ["bbb"]
  processors:
    - rename:
        fields:
           - {from: "message",to: "testmessage"}

采集.log文件

name: hahaha
filebeat.inputs:
- type: log
  enabled: true
  paths: /opt/filebeat-7.6.2-linux-x86_64/test-log.log
  tags: ["bb","aa"]
  fields: 
    ip: 10.192.78.24
output.console:
  pretty: true

测试各个位置的name，fields，tags

name: aaa
tags: ["tag01", "tag02"]
fields: {field01: "001", field02: "002"}
fields_under_root: true
filebeat.inputs:
- type: log
  enabled: true
  paths: /opt/filebeat/filebeat-7.6.2/tmpLogFile/bbb.log
  tags: ["tag03","tag04"]
  fields_under_root: true
  fields: 
    field03: field1
    field04: ${HOSTNAME}
processors:
  - rename:
      fields:
        - {from: "log.file.path", to: "logPath"}
  - drop_fields:
      fields: ["log.file"]
  - add_fields:
      target: "field05"
      fields:
        field005: 005
        fiel006: 006
  - add_tags:
      tags: [tag05, tag06]
      target: "p_tag"
  - add_id: ~
  - fingerprint: 
      fields: ["field01"]
output.console:
  pretty: true
  enable: true

采集包含多行的java程序的.log日志输出到logstash

filebeat.inputs:
- type: log
  enabled: true
  paths: /opt/filebeat/filebeat-7.6.2/tmpLogFile/bbb.log
  multiline.pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
  multiline.negate: false
  multiline.match: after
  fields_under_root: true
  tags: ["aa","bb"]
  fields: 
     ip: 10.192.77.150
output.logstash:
   hosts: [10.192.77.152:5045]