• logstash使用“|”切分日志报错:Exception caught while applying mutate filter {:exception=>"Invalid FieldReference: `message[1]`"}


      如题,通过FileBeat输入的日志里有“|”分割,希望拿到“|”切分后的字段,在logstash配置文件中配置:

    input{
     beats {
        port => "5044"
      }
    }
    
    filter{
        mutate {
          split => {"message"=>"|"}
        }
        
        mutate {
            add_field => {
                "helloObject" => "%{message[1]}"
            }
        }        
              
    }
    output {
      stdout { codec => rubydebug }
    }

      跑logstash和filebeat后,报错:

    D:elklogstash-7.9.0in>.logstash -f ..configlogstash-simple.conf
    Sending Logstash logs to D:/elk/logstash-7.9.0/logs which is now configured via log4j2.properties
    [2020-09-04T07:50:21,150][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 Java HotSpot(TM) 64-Bit Server VM 25.102-b14 on 1.8.0_102-b14 +indy +jit [mswin32-x86_64]"}
    [2020-09-04T07:50:21,545][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
    [2020-09-04T07:50:25,367][INFO ][org.reflections.Reflections] Reflections took 62 ms to scan 1 urls, producing 22 keys and 45 values
    [2020-09-04T07:50:25,656][WARN ][org.logstash.netty.SslContextBuilder] JCE Unlimited Strength Jurisdiction Policy not installed - max key length is 128 bits
    [2020-09-04T07:50:28,752][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/elk/logstash-7.9.0/config/logstash-simple.conf"], :thread=>"#<Thread:0x464291c run>"}
    [2020-09-04T07:50:30,253][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.45}
    [2020-09-04T07:50:30,294][INFO ][logstash.inputs.beats    ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
    [2020-09-04T07:50:30,341][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
    [2020-09-04T07:50:30,568][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
    [2020-09-04T07:50:30,661][INFO ][org.logstash.beats.Server][main][9ae67b9bba079922e3c275620c84df564808b8c801bdc106c4c8bcd0f4d79ee1] Starting server on port: 5044
    [2020-09-04T07:50:31,325][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
    [2020-09-04T07:50:37,539][WARN ][logstash.filters.mutate  ][main][a1eb5d49f96a653afa0c4411fcca0ea2eba55829612774f3236630293da944e0] Exception caught while applying mutate filter {:exception=>"Invalid FieldReference: `message[1]`"}
    D:elklogstash-7.9.0in>.logstash -f ..configlogstash-simple.conf
    Sending Logstash logs to D:/elk/logstash-7.9.0/logs which is now configured via log4j2.properties
    [2020-09-04T07:50:21,150][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 Java HotSpot(TM) 64-Bit Server VM 25.102-b14 on 1.8.0_102-b14 +indy +jit [mswin32-x86_64]"}
    [2020-09-04T07:50:21,545][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
    [2020-09-04T07:50:25,367][INFO ][org.reflections.Reflections] Reflections took 62 ms to scan 1 urls, producing 22 keys and 45 values
    [2020-09-04T07:50:25,656][WARN ][org.logstash.netty.SslContextBuilder] JCE Unlimited Strength Jurisdiction Policy not installed - max key length is 128 bits
    [2020-09-04T07:50:28,752][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/elk/logstash-7.9.0/config/logstash-simple.conf"], :thread=>"#<Thread:0x464291c run>"}
    [2020-09-04T07:50:30,253][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.45}
    [2020-09-04T07:50:30,294][INFO ][logstash.inputs.beats    ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
    [2020-09-04T07:50:30,341][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
    [2020-09-04T07:50:30,568][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
    [2020-09-04T07:50:30,661][INFO ][org.logstash.beats.Server][main][9ae67b9bba079922e3c275620c84df564808b8c801bdc106c4c8bcd0f4d79ee1] Starting server on port: 5044
    [2020-09-04T07:50:31,325][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
    [2020-09-04T07:50:37,539][WARN ][logstash.filters.mutate  ][main][a1eb5d49f96a653afa0c4411fcca0ea2eba55829612774f3236630293da944e0] Exception caught while applying mutate filter {:exception=>"Invalid FieldReference: `message[1]`"}

      虽然切是切了,但想要的日志字段没有加出来:

    {
               "log" => {
              "file" => {
                "path" => "D:\wlf\logs\hello-2020-09-04.0.log"
            },
            "offset" => 111785
        },
               "ecs" => {
            "version" => "1.5.0"
        },
              "tags" => [
            [0] "beats_input_codec_plain_applied",
            [1] "_mutate_error"
        ],
             "input" => {
            "type" => "log"
        },
        "@timestamp" => 2020-09-03T23:48:37.101Z,
             "agent" => {
            "ephemeral_id" => "e3f854ff-d2a1-41fc-9168-ca92bd7dd715",
                    "type" => "filebeat",
                 "version" => "7.9.0",
                "hostname" => "wulf00",
                    "name" => "wulf00",
                      "id" => "ae375dc0-d6e2-488c-be87-2544c05b1242"
        },
              "host" => {
            "name" => "wulf00"
        },
           "message" => [
            [0] "07:48:36.452 [scheduling-1] [] [] INFO  com.wlf.elasticsearchstatictis.Begin - hello",
            [1] "world."
        ],
          "@version" => "1"
    }

      配置文件没问题,但添加字段那一个mutate需要给message套一层中括号:

    mutate {
            add_field => {
                "helloObject" => "%{[message][1]}"
            }
    }

      重新启动logstash,报错没了,message[1]也能取到了:

    D:elklogstash-7.9.0in>.logstash -f ..configlogstash-simple.conf
    Sending Logstash logs to D:/elk/logstash-7.9.0/logs which is now configured via log4j2.properties
    [2020-09-04T08:20:10,122][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.9.0", "jruby.version"=>"jruby 9.2.12.0 (2.5.7) 2020-07-01 db01a49ba6 Java HotSpot(TM) 64-Bit Server VM 25.102-b14 on 1.8.0_102-b14 +indy +jit [mswin32-x86_64]"}
    [2020-09-04T08:20:10,584][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
    [2020-09-04T08:20:13,684][INFO ][org.reflections.Reflections] Reflections took 45 ms to scan 1 urls, producing 22 keys and 45 values
    [2020-09-04T08:20:13,937][WARN ][org.logstash.netty.SslContextBuilder] JCE Unlimited Strength Jurisdiction Policy not installed - max key length is 128 bits
    [2020-09-04T08:20:16,047][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/elk/logstash-7.9.0/config/logstash-simple.conf"], :thread=>"#<Thread:0x3bb3812e run>"}
    [2020-09-04T08:20:17,233][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.16}
    [2020-09-04T08:20:17,268][INFO ][logstash.inputs.beats    ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
    [2020-09-04T08:20:17,326][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
    [2020-09-04T08:20:17,536][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
    [2020-09-04T08:20:17,607][INFO ][org.logstash.beats.Server][main][be332a0e557bc16ff2fcdf24fcbd3b24c16b1141269fff75b0e835137ee7698f] Starting server on port: 5044
    [2020-09-04T08:20:18,205][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
    {
               "host" => {
            "name" => "wulf00"
        },
                "ecs" => {
            "version" => "1.5.0"
        },
               "tags" => [
            [0] "beats_input_codec_plain_applied"
        ],
        "helloObject" => "world.",
         "@timestamp" => 2020-09-04T00:20:14.782Z,
              "agent" => {
                "hostname" => "wulf00",
            "ephemeral_id" => "e3f854ff-d2a1-41fc-9168-ca92bd7dd715",
                 "version" => "7.9.0",
                      "id" => "ae375dc0-d6e2-488c-be87-2544c05b1242",
                    "type" => "filebeat",
                    "name" => "wulf00"
        },
            "message" => [
            [0] "08:20:11.653 [scheduling-1] [] [] INFO  com.wlf.elasticsearchstatictis.Begin - hello",
            [1] "world."
        ],
           "@version" => "1",
                "log" => {
            "offset" => 131366,
              "file" => {
                "path" => "D:\wlf\logs\hello-2020-09-04.0.log"
            }
        },
              "input" => {
            "type" => "log"
        }
    }
  • 相关阅读:
    描述一下 JVM 加载 class 文件的原理机制?
    Java 中会存在内存泄漏吗,请简单描述
    关于同步机制的一些见解
    Mybatis 一对一,一对多,多对一,多对多的理解
    关于JavaBean实现Serializable接口的见解
    Python 文件I/O
    Python 模块
    Python 函数
    Python time tzset()方法
    Python time time()方法
  • 原文地址:https://www.cnblogs.com/wuxun1997/p/13611639.html
Copyright © 2020-2023  润新知