0x00 概述
ELK-logstash在搬运日志的时候会出现多行日志，普通的搬运会造成保存到ES中单条单条，很丑，而且不方便读取，logstash-filter-multiline可以解决该问题
github地址：https://github.com/logstash-plugins/logstash-filter-multiline
其他插件的地址：https://github.com/logstash-plugins
官网地址：https://www.elastic.co/cn/products/logstash
普通日志如下：
记录到es会的记录则是：
我们希望的结果肯定是这样的
0x01 原生安装
# /usr/share/logstash/bin/logstash-plugin install logstash-filter-multiline Validating logstash-filter-multiline Installing logstash-filter-multiline WARNING: can not set Session#timeout=(0) no session context
0x02 改镜像源安装
给 Ruby 加上国内的镜像站：https://gems.ruby-china.com/，替代https://rubygems.org。
注意原https://gems.ruby-china.org/目前已经可用，需要使用https://gems.ruby-china.com/
1. 安装Gem并更新
# yum install -y gem # gem -v 2.0.14.1 # gem update --system # gem -v 2.6.13
2. 检查并修改镜像源
# gem sources -l *** CURRENT SOURCES *** https://rubygems.org/
# gem sources --add https://gems.ruby-china.com/ --remove https://rubygems.org/ https://gems.ruby-china.com/ added to sources https://rubygems.org/ removed from sources
# cat ~/.gemrc --- :backtrace: false :bulk_threshold: 1000 :sources: - https://gems.ruby-china.com/ :update_sources: true :verbose: true
3. 安装 bundle 并更改镜像源
# gem install bundler # bundle config mirror.https://rubygems.org https://gems.ruby-china.com
4. 修改 logstash的 gem 镜像源
# vim /usr/share/logstash/Gemfile # This is a Logstash generated Gemfile. # If you modify this file manually all comments and formatting will be lost. source "https://rubygems.org" gem "logstash-core", :path => "./logstash-core" ......
# 更改默认的 https://rubygems.org 为https://gems.ruby-china.com
Gemfile.jruby-1.9.lock文件不用改，会自动更新。
5. 安装 logstash-filter-multiline
# /usr/share/logstash/bin/logstash-plugin install logstash-filter-multiline Validating logstash-filter-multiline Installing logstash-filter-multiline Installation successful
我们看看logstash都安装了哪些插件
# /usr/share/logstash/bin/logstash-plugin list logstash-codec-cef logstash-codec-collectd logstash-codec-dots logstash-codec-edn logstash-codec-edn_lines logstash-codec-es_bulk logstash-codec-fluent logstash-codec-graphite logstash-codec-json logstash-codec-json_lines logstash-codec-line logstash-codec-msgpack logstash-codec-multiline
logstash-filter-multiline logstash-codec-netflow logstash-codec-plain logstash-codec-rubydebug logstash-filter-aggregate logstash-filter-anonymize logstash-filter-cidr logstash-filter-clone logstash-filter-csv logstash-filter-date logstash-filter-de_dot logstash-filter-dissect logstash-filter-dns logstash-filter-drop logstash-filter-elasticsearch logstash-filter-fingerprint logstash-filter-geoip logstash-filter-grok logstash-filter-jdbc_static logstash-filter-jdbc_streaming logstash-filter-json logstash-filter-kv logstash-filter-metrics logstash-filter-mutate logstash-filter-ruby logstash-filter-sleep logstash-filter-split logstash-filter-syslog_pri logstash-filter-throttle logstash-filter-translate logstash-filter-truncate logstash-filter-urldecode logstash-filter-useragent logstash-filter-xml logstash-input-beats logstash-input-dead_letter_queue logstash-input-elasticsearch logstash-input-exec logstash-input-file logstash-input-ganglia logstash-input-gelf logstash-input-generator logstash-input-graphite logstash-input-heartbeat logstash-input-http logstash-input-http_poller logstash-input-imap logstash-input-jdbc logstash-input-kafka logstash-input-pipe logstash-input-rabbitmq logstash-input-redis logstash-input-s3 logstash-input-snmptrap logstash-input-sqs logstash-input-stdin logstash-input-syslog logstash-input-tcp logstash-input-twitter logstash-input-udp logstash-input-unix logstash-output-cloudwatch logstash-output-csv logstash-output-elasticsearch logstash-output-email logstash-output-file logstash-output-graphite logstash-output-http logstash-output-kafka logstash-output-lumberjack logstash-output-nagios logstash-output-null logstash-output-pagerduty logstash-output-pipe logstash-output-rabbitmq logstash-output-redis logstash-output-s3 logstash-output-sns logstash-output-sqs logstash-output-stdout logstash-output-tcp logstash-output-udp logstash-output-webhdfs logstash-patterns-core
6. 离线打包
这里安装好了，那么就可以打个离线的 zip 包，方便生产环境直接安装。离线包将包含所有依赖的包。
# cd /usr/share/logstash/bin
# ./logstash-plugin prepare-offline-pack --overwrite --output logstash-filter-multiline.zip logstash-filter-multiline
Offline package created at: logstash-filter-multiline.zip
You can install it with this command `bin/logstash-plugin install file:///usr/share/logstash/bin/logstash-filter-multiline.zip`
7. logstash-filter-multiline使用方法
codec =>multiline {
charset=>... #可选 字符编码
max_bytes=>... #可选 bytes类型 设置最大的字节数
max_lines=>... #可选 number类型 设置最大的行数,默认是500行
multiline_tag... #可选 string类型 设置一个事件标签,默认是multiline
pattern=>... #必选 string类型 设置匹配的正则表达式
patterns_dir=>... #可选 array类型 可以设置多个正则表达式
negate=>... #可选 boolean类型 默认false不显示，可设置ture
what=>... #必选 向前previous ， 向后 next
}
## negate 只支持布尔值，true 或者false，默认为false。 如果设置为true，表示与正则表达式（pattern）不匹配的内容都需要整合， 具体整合在前还是在后，看what参数。如果设置为false，即与pattern匹配的内容 ## what 前一行 或者后一行，指出上面对应的规则与前一行内容收集为一行，还是与后一行整合在一起 negate默认是 false，不显示与patten匹配的行 由what决定 向前或向后 匹配 negate 设置为true 则与patten 不匹配的行 由what决定 向前或向后 匹配