如题,通过FileBeat输入的日志里有“|”分割,希望拿到“|”切分后的字段,在logstash配置文件中配置:
input{ beats { port => "5044" } } filter{ mutate { split => {"message"=>"|"} } mutate { add_field => { "helloObject" => "%{message[1]}" } } } output { stdout { codec => rubydebug } }
跑logstash和filebeat后,报错:
D:elklogstash-7.9.0in>.logstash -f ..configlogstash-simple.conf Sending Logstash logs to D:/elk/logstash-7.9.0/logs which is now configured via log4j2.properties [2020-09-04T07:50:21,150][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 Java HotSpot(TM) 64-Bit Server VM 25.102-b14 on 1.8.0_102-b14 +indy +jit [mswin32-x86_64]"} [2020-09-04T07:50:21,545][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-09-04T07:50:25,367][INFO ][org.reflections.Reflections] Reflections took 62 ms to scan 1 urls, producing 22 keys and 45 values [2020-09-04T07:50:25,656][WARN ][org.logstash.netty.SslContextBuilder] JCE Unlimited Strength Jurisdiction Policy not installed - max key length is 128 bits [2020-09-04T07:50:28,752][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/elk/logstash-7.9.0/config/logstash-simple.conf"], :thread=>"#<Thread:0x464291c run>"} [2020-09-04T07:50:30,253][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.45} [2020-09-04T07:50:30,294][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"} [2020-09-04T07:50:30,341][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"} [2020-09-04T07:50:30,568][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2020-09-04T07:50:30,661][INFO ][org.logstash.beats.Server][main][9ae67b9bba079922e3c275620c84df564808b8c801bdc106c4c8bcd0f4d79ee1] Starting server on port: 5044 [2020-09-04T07:50:31,325][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2020-09-04T07:50:37,539][WARN ][logstash.filters.mutate ][main][a1eb5d49f96a653afa0c4411fcca0ea2eba55829612774f3236630293da944e0] Exception caught while applying mutate filter {:exception=>"Invalid FieldReference: `message[1]`"} D:elklogstash-7.9.0in>.logstash -f ..configlogstash-simple.conf Sending Logstash logs to D:/elk/logstash-7.9.0/logs which is now configured via log4j2.properties [2020-09-04T07:50:21,150][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 Java HotSpot(TM) 64-Bit Server VM 25.102-b14 on 1.8.0_102-b14 +indy +jit [mswin32-x86_64]"} [2020-09-04T07:50:21,545][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-09-04T07:50:25,367][INFO ][org.reflections.Reflections] Reflections took 62 ms to scan 1 urls, producing 22 keys and 45 values [2020-09-04T07:50:25,656][WARN ][org.logstash.netty.SslContextBuilder] JCE Unlimited Strength Jurisdiction Policy not installed - max key length is 128 bits [2020-09-04T07:50:28,752][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/elk/logstash-7.9.0/config/logstash-simple.conf"], :thread=>"#<Thread:0x464291c run>"} [2020-09-04T07:50:30,253][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.45} [2020-09-04T07:50:30,294][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"} [2020-09-04T07:50:30,341][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"} [2020-09-04T07:50:30,568][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2020-09-04T07:50:30,661][INFO ][org.logstash.beats.Server][main][9ae67b9bba079922e3c275620c84df564808b8c801bdc106c4c8bcd0f4d79ee1] Starting server on port: 5044 [2020-09-04T07:50:31,325][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2020-09-04T07:50:37,539][WARN ][logstash.filters.mutate ][main][a1eb5d49f96a653afa0c4411fcca0ea2eba55829612774f3236630293da944e0] Exception caught while applying mutate filter {:exception=>"Invalid FieldReference: `message[1]`"}
虽然切是切了,但想要的日志字段没有加出来:
{ "log" => { "file" => { "path" => "D:\wlf\logs\hello-2020-09-04.0.log" }, "offset" => 111785 }, "ecs" => { "version" => "1.5.0" }, "tags" => [ [0] "beats_input_codec_plain_applied", [1] "_mutate_error" ], "input" => { "type" => "log" }, "@timestamp" => 2020-09-03T23:48:37.101Z, "agent" => { "ephemeral_id" => "e3f854ff-d2a1-41fc-9168-ca92bd7dd715", "type" => "filebeat", "version" => "7.9.0", "hostname" => "wulf00", "name" => "wulf00", "id" => "ae375dc0-d6e2-488c-be87-2544c05b1242" }, "host" => { "name" => "wulf00" }, "message" => [ [0] "07:48:36.452 [scheduling-1] [] [] INFO com.wlf.elasticsearchstatictis.Begin - hello", [1] "world." ], "@version" => "1" }
配置文件没问题,但添加字段那一个mutate需要给message套一层中括号:
mutate { add_field => { "helloObject" => "%{[message][1]}" } }
重新启动logstash,报错没了,message[1]也能取到了:
D:elklogstash-7.9.0in>.logstash -f ..configlogstash-simple.conf Sending Logstash logs to D:/elk/logstash-7.9.0/logs which is now configured via log4j2.properties [2020-09-04T08:20:10,122][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 Java HotSpot(TM) 64-Bit Server VM 25.102-b14 on 1.8.0_102-b14 +indy +jit [mswin32-x86_64]"} [2020-09-04T08:20:10,584][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-09-04T08:20:13,684][INFO ][org.reflections.Reflections] Reflections took 45 ms to scan 1 urls, producing 22 keys and 45 values [2020-09-04T08:20:13,937][WARN ][org.logstash.netty.SslContextBuilder] JCE Unlimited Strength Jurisdiction Policy not installed - max key length is 128 bits [2020-09-04T08:20:16,047][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/elk/logstash-7.9.0/config/logstash-simple.conf"], :thread=>"#<Thread:0x3bb3812e run>"} [2020-09-04T08:20:17,233][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.16} [2020-09-04T08:20:17,268][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"} [2020-09-04T08:20:17,326][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"} [2020-09-04T08:20:17,536][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2020-09-04T08:20:17,607][INFO ][org.logstash.beats.Server][main][be332a0e557bc16ff2fcdf24fcbd3b24c16b1141269fff75b0e835137ee7698f] Starting server on port: 5044 [2020-09-04T08:20:18,205][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} { "host" => { "name" => "wulf00" }, "ecs" => { "version" => "1.5.0" }, "tags" => [ [0] "beats_input_codec_plain_applied" ], "helloObject" => "world.", "@timestamp" => 2020-09-04T00:20:14.782Z, "agent" => { "hostname" => "wulf00", "ephemeral_id" => "e3f854ff-d2a1-41fc-9168-ca92bd7dd715", "version" => "7.9.0", "id" => "ae375dc0-d6e2-488c-be87-2544c05b1242", "type" => "filebeat", "name" => "wulf00" }, "message" => [ [0] "08:20:11.653 [scheduling-1] [] [] INFO com.wlf.elasticsearchstatictis.Begin - hello", [1] "world." ], "@version" => "1", "log" => { "offset" => 131366, "file" => { "path" => "D:\wlf\logs\hello-2020-09-04.0.log" } }, "input" => { "type" => "log" } }