通过fluentd配置输出到elasticsearch，启动服务报错 Using Elasticsearch client 8.4.0 is not compatible for your Elasticsearch server

问题描述

 

通过fluentd的配置文件，将fluentd的输出定向到elasticsearch中，配置文件如下：

 

<match syslog.**>
  @type elasticsearch
  host 172.20.58.152
  port 19200
  logstash_format true
  <buffer>
    flush_interval 10s # for testing
  </buffer>
</match>

 

 

发现在启动的时候报错：

 

2022-11-14 03:51:59 -0500 [info]: starting fluentd-1.15.3 pid=18599 ruby="2.7.6"
2022-11-14 03:51:59 -0500 [info]: spawn command to main:  cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2022-11-14 03:51:59 -0500 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2022-11-14 03:52:00 -0500 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2022-11-14 03:52:00 -0500 [info]: adding match pattern="syslog.**" type="elasticsearch"
2022-11-14 03:52:00 -0500 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.4.0 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server."
2022-11-14 03:52:00 -0500 [error]: Worker 0 finished unexpectedly with status 2
2022-11-14 03:52:00 -0500 [info]: Received graceful stop

 

问题分析

 

通过报错和提示看，应该是fluentd的elastichsearch插件的版本和elasticsearch server的版本不匹配

 

查看fluentd中elasticsearch中插件的版本

 

发现，正好是报错提示中的 8.4.0 版本

 

[root@centos7 gems]# td-agent-gem list | grep elastic
elastic-transport (8.1.0)
elasticsearch (8.4.0)
elasticsearch-api (8.4.0)
fluent-plugin-elasticsearch (5.2.4)

 

查看elasticsearch的版本是：6.1.0

 

[elasticsearch@nctest-snap-test-02 bin]$ ./elasticsearch --version
Version: 6.1.0, Build: c0c1ba0/2017-12-12T12:32:54.550Z, JVM: 1.8.0_202
[elasticsearch@nctest-snap-test-02 bin]$ 

 

所以，使用高版本的客户端连接低版本的elasticsearch server有版本兼容性的问题！

 

所以，解决的方法也非常的简单

 

问题解决

 

1、卸载elasticsearch(8.4.0)插件

 

td-agent-gem uninstall elasticsearch
td-agent-gem uninstall elasticsearch-api-8.4.0

 

2、安装和elasticsearch server一样版本的插件

 

td-agent-gem install elasticsearch -v 6.1.0

 

安装之后，确认插件版本已经安装正确

 

[root@centos7 gems]# td-agent-gem list | grep elastic
elastic-transport (8.1.0)
elasticsearch (6.1.0)
elasticsearch-api (6.1.0)
elasticsearch-transport (6.1.0)
fluent-plugin-elasticsearch (5.2.4)

 

3、重新启动fluentd服务

 

systemctl restart td-agent

问题搞定！