使用 filebeat原生处理日志时间，就是使用日志文件中message字段值开头的时间覆盖默认的@timestamp时间，使用filebeat processor配置，pipeline不符合要求

情况说明

Filebeat 收集的日志发送到 ElasticSearch 后，会默认添加一个 @timestamp 字段做为时间戳用于检索，而日志中的信息会所有添加到 message 字段中，可是这个时间是 Filebeat 采集日志的时间，不是日志生成的实际时间，因此为了便于检索日志，须要将 @timestamp 替换为 message 字段中的时间。

首先日志格式以下：

# info文件的日志内容

2022-05-24 00:12:19.196 INFO  com.jdd.parking.modules.chargefee.standard.rest.GetCarChargeFeeController Line:131 - 有牌车出口码计费,plateNo:鄂A98J51
2022-05-24 00:12:19.196 INFO  com.jdd.parking.modules.chargefee.standard.service.impl.ChargeFeeV2ServiceImpl Line:64  - 计费服务类,出场码计费,传入参数:OutParkChargeFeeDTO(parkCode=58818, serialNo=c79e5114-7103aa77, plateNo=鄂A98J51, imgPath=https://jdd-parking.oss-cn-qingdao.aliyuncs.com/picture/1/c79e5114-7103aa77/20220524/00/20220524_001218_813.jpg)
2022-05-24 00:12:19.205 INFO  com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils Line:444 - 计费服务类,计费开始,车牌号：鄂A98J51,openId:null
2022-05-24 00:12:19.208 INFO  com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils Line:877 - 计费服务类,单次入场纪录计费,传入参数：计费区域Id：59a3ef3754d87c85b02c51e4ed0e03ae,计费开始时间：2022-05-23 22:44:04,计费结束时间：2022-05-24 00:12:19
2022-05-24 00:12:19.209 INFO  com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils Line:897 - 计费服务类,单次入场纪录计费,按24小时计算一天,停车整天数：0天,不足整天的停车时间：88分钟

# error文件的日志内容

2022-05-24 00:12:19.210 ERROR com.jdd.parking.modules.chargefee.standard.service.impl.ChargeFeeV2ServiceImpl Line:129 - 计费服务类,异常信息
com.jdd.parking.common.exception.JddException: 计费规则模块不存在
	at com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils.lessThanOneDayFee(ChargeFeeUtils.java:85)
	at com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils.chargeEnterLogFee(ChargeFeeUtils.java:920)
	at com.jdd.parking.modules.chargefee.standard.utils.ChargeFeeUtils.chargeTotalFeeMonthlyV3(ChargeFeeUtils.java:525)
	at com.jdd.parking.modules.chargefee.standard.service.impl.ChargeFeeV2ServiceImpl.outParkChargeFeeV2(ChargeFeeV2ServiceImpl.java:97)
	at com.jdd.parking.modules.chargefee.standard.rest.GetCarChargeFeeController.getOutParkChargeFeeV2(GetCarChargeFeeController.java:133)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)
	at java.lang.Thread.run(Thread.java:748)
2022-05-24 00:12:19.211 ERROR com.jdd.parking.modules.chargefee.standard.rest.GetCarChargeFeeController Line:137 - 有牌车出口码计费失败,错误信息：{}
com.jdd.parking.common.exception.JddException: 计费服务类,出场码计费系统错误,计费规则模块不存在
	at com.jdd.parking.modules.chargefee.standard.service.impl.ChargeFeeV2ServiceImpl.outParkChargeFeeV2(ChargeFeeV2ServiceImpl.java:130)
	at com.jdd.parking.modules.chargefee.standard.rest.GetCarChargeFeeController.getOutParkChargeFeeV2(GetCarChargeFeeController.java:133)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)
	at java.lang.Thread.run(Thread.java:748)

当前的日志传输路径是filebeat->es，没有使用logstash

解决办法

使用elasticsearch pipeline功能

参考文章：http://www.javashuo.com/article/p-ghefxmal-nu.html

索引文件名上的时间是当前时间

pipeline中没有加"timezone": "Asia/Shanghai", 可以看到，有的是当前系统时间，有的是UTC时间，但是真正想要的时间是日志message开头的时间

pipeline中加上"timezone": "Asia/Shanghai", 可以看到，有的是当前系统时间，有的是UTC时间，但是真正想要的时间是日志message开头的时间

结合以上实践，使用elasticsearch pipeline功能不符合要求，建议使用使用filebeat processor配置

使用filebeat processor配置

参考网址：https://blog.csdn.net/qq_34083066/article/details/115354511

索引文件名上的时间是日志message开头截取的时间

# cat /etc/filebeat/filebeat.yml
filebeat.inputs:

- type: log
  enabled: true
  paths:
    - /opt/log_error.log
  fields:
    log_source: fee
    log_type: error
  fields_under_root: true
  tags: ["application"]
  multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d* ' # 匹配日志开头的时间，划分为一行
  multiline.negate: true
  multiline.match: after

- type: log
  enabled: true
  paths:
    - /opt/log_info.log
  fields:
    log_source: fee
    log_type: info
  fields_under_root: true
  tags: ["application"]
  multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d* '
  multiline.negate: true
  multiline.match: after

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  reload.period: 10s

setup.ilm.enabled: false
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0
  index.codec: best_compression

setup.dashboards.enabled: true

setup.kibana:
  host: "192.168.20.250:5601"

output.elasticsearch:
  hosts: ["localhost:9200"]
  indices:
    - index: "work-%{+yyyy.MM.dd}"
      when.contains:
        tags: "application"

processors:
  - decode_json_fields:
      fields: ["message"]
      target: "" 
      overwrite_keys: false
      process_array: false
      max_depth: 1
  # 这个脚本的内容是根据message字段值开始的俩空格截取值然后拼接在一起，也就是时间：2022-05-24 00:12:19.211，赋给新变量：start_time
  - script:
      lang: javascript
      id: my_filter
      tag: enable
      source: >
        function process(event) {
            var str= event.Get("message");
            var time =str.split(" ").slice(0,2).join(" ");
            event.Put("start_time",time);
        }
  # 使用新变量的值start_time替换默认的@timestamp的值
  - timestamp:
      field: start_time
      #timezone: Asia/Shanghai
      target_field: "@timestamp"
      layouts:
        - '2006-01-02 15:04:05'
        - '2006-01-02 15:04:05.999'
      test:
        - '2006-01-02 15:04:05'
        - '2006-01-02 15:04:05.999'
  # 移除无用的字段
  - drop_fields:
      fields: ["input","agent","ecs","start_time"]

效果显示

本文中timezone说明

@timestamp 这个字段是系统自带的，默认时间就是采集log的UTC时间 。例如 现在时间是2020-12-30 18:59:58.234 ，那么他的值为2020-12-30T10:59:58.234Z

现在我们手动替换@timestamp为日志时间（东八区），需要将时间字符串解析成UTC时间，赋值给@timestamp，那么我们必须显示的指定timezone 信息。

1.指定timezone后的表现
原先log时间字符串：2020-08-05 16:21:51.824。重新赋值的@timestamp值： 2020-08-05T08:21:51.824Z （正确）

2.未指定timezone后的表现

原先log时间字符串：2020-08-05 16:21:51.824。重新赋值的@timestamp值： 2020-08-05T21:51.824Z （默认使用了UTC零时区解析）

按上面配置是没有指定timezone的，这个一定要注意。若是指定了则会跟log时间相差8个小时