Elastic Stack之 Logstash 6.7.1版本安装

1、截至目前Elasticsearch 版本已经更新到了7.10.1版本了，这里先使用Logstash 6.7.1版本，给一个下载地址，如下所示：

官方下载地址：https://www.elastic.co/cn/downloads/past-releases#elasticsearch

2、Logstash入门，简介data shipper （不是轻量级的，会比beats占用更多的资源，但是功能强大）。

　　a）、ETL的概念：Extract 对数据进行提取、Transform 转换、Load 对外的输出。

　　b）、Logstash 是一个开源的，服务端的数据处理流，可以同时从多个数据源提取数据、转换数据、最后把数据放到你要存储的地方。

3、Logstash处理流程，如下所示：

　　a）、input：可以从file 、Redis 、beats（filebeats等等beats）、kafka等读取数据。

处理流程，Input和Output的配置，由于Logstash不是yaml语法。
    input{file{path => "/tmp/abc.log"}},案例一
    output{stdout{codec => rubydebug}}，案例二

　　b）、filter ：支持gork(表达式，简单理解为基于正则的，可以将非格式化数据转化成格式化数据的语法)、mutate（可以对结构化的数据的字段进行增删改查）、drop、date。

处理流程，Filter配置。
    Grok，基于正则表达式提供了丰富可重用的模式（pattern）。基于此可以将非结构化数据做结构化处理。
    Date，将字符串类型的时间字段转换为时间戳类型，方便后续数据处理。
    Mutate，进行增加，修改，删除，替换等字段相关的处理。

　　c）、output ：可以向stdout 、elasticsearch 、Redis、kafka等中输出数据。

4、将下载好的logstash（Logstash是Ruby开发的哦）安装包上传到服务器，进行解压缩，然后授权给elsearch用户，如下所示：

[elsearch@k8s-master package]# tar -zxvf logstash-6.7.1.tar.gz -C /usr/local/elastic/

[root@k8s-master elastic]# ll
total 0
drwxr-xr-x  9 elsearch elsearch 155 Jan  9 23:08 elasticsearch-6.7.1
drwxr-xr-x  6 elsearch elsearch 241 Jan 10 20:05 filebeat-6.7.1-linux-x86_64
drwxr-xr-x 13 elsearch elsearch 263 Jan  9 23:41 kibana-6.7.1-linux-x86_64
drwxr-xr-x 12 root     root     255 Jan 10 20:31 logstash-6.7.1
[root@k8s-master elastic]# chown -R elsearch:elsearch logstash-6.7.1/
[root@k8s-master elastic]# ll
total 0
drwxr-xr-x  9 elsearch elsearch 155 Jan  9 23:08 elasticsearch-6.7.1
drwxr-xr-x  6 elsearch elsearch 241 Jan 10 20:05 filebeat-6.7.1-linux-x86_64
drwxr-xr-x 13 elsearch elsearch 263 Jan  9 23:41 kibana-6.7.1-linux-x86_64
drwxr-xr-x 12 elsearch elsearch 255 Jan 10 20:31 logstash-6.7.1
[root@k8s-master elastic]# 

此处还是使用logstash来收集nginx日志，如下所示：

[root@k8s-master logstash-6.7.1]# head -n 2 /var/log/nginx/access.log 
192.168.110.1 - - [21/Jul/2019:21:52:34 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36"
192.168.110.1 - - [21/Jul/2019:21:52:34 +0800] "GET /favicon.ico HTTP/1.1" 404 570 "http://192.168.110.133/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36"
[root@k8s-master logstash-6.7.1]# ll
total 848
drwxr-xr-x  2 elsearch elsearch   4096 Jan 10 20:31 bin
drwxr-xr-x  2 elsearch elsearch    142 Jan 10 20:31 config
-rw-r--r--  1 elsearch elsearch   2276 Apr  3  2019 CONTRIBUTORS
drwxr-xr-x  2 elsearch elsearch      6 Apr  3  2019 data
-rw-r--r--  1 elsearch elsearch   4194 Apr  3  2019 Gemfile
-rw-r--r--  1 elsearch elsearch  22455 Apr  3  2019 Gemfile.lock
drwxr-xr-x  6 elsearch elsearch     84 Jan 10 20:31 lib
-rw-r--r--  1 elsearch elsearch  13675 Apr  3  2019 LICENSE.txt
drwxr-xr-x  4 elsearch elsearch     90 Jan 10 20:31 logstash-core
drwxr-xr-x  3 elsearch elsearch     86 Jan 10 20:31 logstash-core-plugin-api
drwxr-xr-x  4 elsearch elsearch     55 Jan 10 20:31 modules
-rw-r--r--  1 elsearch elsearch 808305 Apr  3  2019 NOTICE.TXT
drwxr-xr-x  3 elsearch elsearch     30 Jan 10 20:31 tools
drwxr-xr-x  4 elsearch elsearch     33 Jan 10 20:31 vendor
drwxr-xr-x 10 elsearch elsearch    205 Jan 10 20:31 x-pack
[root@k8s-master logstash-6.7.1]# cd config/
[root@k8s-master config]# ll
total 36
-rw-r--r-- 1 elsearch elsearch 1829 Apr  3  2019 jvm.options
-rw-r--r-- 1 elsearch elsearch 4568 Apr  3  2019 log4j2.properties
-rw-r--r-- 1 elsearch elsearch  342 Apr  3  2019 logstash-sample.conf
-rw-r--r-- 1 elsearch elsearch 8204 Apr  3  2019 logstash.yml
-rw-r--r-- 1 elsearch elsearch 3244 Apr  3  2019 pipelines.yml
-rw-r--r-- 1 elsearch elsearch 1696 Apr  3  2019 startup.options
[root@k8s-master config]# vim logstash.yml 
[root@k8s-master config]# cp logstash-sample.conf nginx-logstash.conf
[root@k8s-master config]# vim nginx-logstash.conf 
[root@k8s-master config]# 

nginx-logstash.conf配置文件，如下所示：

input {
  stdin { }
}

filter {
  grok {
    match => {
      "message" => '%{IPORHOST:remote_ip} - %{DATA:user_name} [%{HTTPDATE:time}] "%{WORD:request_action} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:response} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:agent}"'
    }
  }

  date {
    match => [ "time", "dd/MMM/YYYY:HH:mm:ss Z" ]
    locale => en
  }

  geoip {
    source => "remote_ip"
    target => "geoip"
  }

  useragent {
    source => "agent"
    target => "user_agent"
  }
}

output {
stdout {
 codec => rubydebug 
 }
}

启动，正常情况，如下所示：

[elsearch@k8s-master logstash-6.7.1]$ head -n 2 /var/log/nginx/access.log | ./bin/logstash -f config/nginx-logstash.conf 
Sending Logstash logs to /usr/local/elastic/logstash-6.7.1/logs which is now configured via log4j2.properties
[2021-01-10T21:09:04,032][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-01-10T21:09:04,050][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.7.1"}
[2021-01-10T21:09:14,231][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2021-01-10T21:09:14,592][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2021-01-10T21:09:15,316][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7eea5747 run>"}
[2021-01-10T21:09:15,470][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-01-10T21:09:16,380][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
          "referrer" => "-",
              "host" => "k8s-master",
          "response" => "200",
              "tags" => [
        [0] "_geoip_lookup_failure"
    ],
        "@timestamp" => 2019-07-21T13:52:34.000Z,
         "remote_ip" => "192.168.110.1",
             "agent" => "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36",
           "message" => "192.168.110.1 - - [21/Jul/2019:21:52:34 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36"",
        "user_agent" => {
             "os" => "Windows",
          "build" => "",
          "major" => "74",
        "os_name" => "Windows",
         "device" => "Other",
          "patch" => "3729",
           "name" => "Chrome",
          "minor" => "0"
    },
         "user_name" => "-",
    "request_action" => "GET",
           "request" => "/",
             "geoip" => {},
          "@version" => "1",
              "time" => "21/Jul/2019:21:52:34 +0800",
             "bytes" => "612",
      "http_version" => "1.1"
}
{
          "referrer" => "http://192.168.110.133/",
              "host" => "k8s-master",
          "response" => "404",
              "tags" => [
        [0] "_geoip_lookup_failure"
    ],
        "@timestamp" => 2019-07-21T13:52:34.000Z,
         "remote_ip" => "192.168.110.1",
             "agent" => "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36",
           "message" => "192.168.110.1 - - [21/Jul/2019:21:52:34 +0800] "GET /favicon.ico HTTP/1.1" 404 570 "http://192.168.110.133/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36"",
        "user_agent" => {
             "os" => "Windows",
          "build" => "",
          "major" => "74",
        "os_name" => "Windows",
         "device" => "Other",
          "patch" => "3729",
           "name" => "Chrome",
          "minor" => "0"
    },
         "user_name" => "-",
    "request_action" => "GET",
           "request" => "/favicon.ico",
             "geoip" => {},
          "@version" => "1",
              "time" => "21/Jul/2019:21:52:34 +0800",
             "bytes" => "570",
      "http_version" => "1.1"
}
[2021-01-10T21:09:16,618][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x7eea5747 run>"}
[elsearch@k8s-master logstash-6.7.1]$ 

如果报错，那估计就是自己的conf配置文件，比如格式，还是拼写，出现问题了，如下所示：

[elsearch@k8s-master logstash-6.7.1]$ head -n 2 /var/log/nginx/access.log | ./bin/logstash -f config/nginx-logstash.conf 
Sending Logstash logs to /usr/local/elastic/logstash-6.7.1/logs which is now configured via log4j2.properties
[2021-01-10T21:02:50,780][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-01-10T21:02:50,800][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.7.1"}
[2021-01-10T21:03:02,953][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2021-01-10T21:03:03,310][ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::FilterDelegator:0x12758a4d>", :error=>"pattern %{HTTPDATA:time} not defined", :thread=>"#<Thread:0x206ac3e9 run>"}
[2021-01-10T21:03:03,329][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{HTTPDATA:time} not defined>, :backtrace=>["/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in `block in compile'", "org/jruby/RubyKernel.java:1411:in `loop'", "/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in `compile'", "/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:281:in `block in register'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:275:in `block in register'", "org/jruby/RubyHash.java:1419:in `each'", "/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:270:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56:in `register'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:259:in `register_plugin'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:270:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:270:in `register_plugins'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:612:in `maybe_setup_out_plugins'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:280:in `start_workers'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:217:in `run'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/pipeline.rb:176:in `block in start'"], :thread=>"#<Thread:0x206ac3e9 run>"}
[2021-01-10T21:03:03,348][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[elsearch@k8s-master logstash-6.7.1]$ 

如果报下面的错，那是logstash-6.7.1目录下面的data删除掉就行了，或者备份了，我可能是使用root启动了，下次使用自己的账号启动就可以重新自动生成了。

[elsearch@k8s-master logstash-6.7.1]$ head -n 2 /var/log/nginx/access.log | ./bin/logstash -f config/nginx-logstash.conf 
Sending Logstash logs to /usr/local/elastic/logstash-6.7.1/logs which is now configured via log4j2.properties
[2021-01-10T20:56:42,326][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/local/elastic/logstash-6.7.1/data/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/settings.rb:447:in `validate'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/settings.rb:229:in `validate_value'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/settings.rb:140:in `block in validate_all'", "org/jruby/RubyHash.java:1419:in `each'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/settings.rb:139:in `validate_all'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/runner.rb:278:in `execute'", "/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/local/elastic/logstash-6.7.1/logstash-core/lib/logstash/runner.rb:237:in `run'", "/usr/local/elastic/logstash-6.7.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/local/elastic/logstash-6.7.1/lib/bootstrap/environment.rb:73:in `<main>'"]}
[2021-01-10T20:56:42,354][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit