elasticsearch的一些运维和开发常见问题

elasticsearch missing authentication credentials for REST request

现在网上错误的、没有经过验证的技术贴漫天飞，很容易给读者造成各种误导，由于这部分关系，笔者也开始认真地进行技术贴的撰写，同时也mark下一些知识点，特别是在这种由于公司内网环境导致无法把资料迁出的情况。
场景
elasticsearch 在开启xpack认证之后, 直接通过curl 访问接口会报错
解决
curl带上认证
在ECS终端访问REST API的时候
curl --user elastic:yourpassword -XGET ‘localhost:9200’

elastic是默认的账号

权限不够或 can not run elasticsearch as root

问题：

　　es安装好之后，使用root启动会报错：can not run elasticsearch as root

[root@iZbp1bb2egi7w0ueys548pZ bin]# ./elasticsearch
[2019-01-21T09:50:59,387][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:134) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.0.0.jar:6.0.0]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:171) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) ~[elasticsearch-6.0.0.jar:6.0.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) ~[elasticsearch-6.0.0.jar:6.0.0]
    ... 6 more

原因：

　　为了安全不允许使用root用户启动

解决：

　　es5之后的都不能使用添加启动参数或者修改配置文件等方法启动了，必须要创建用户

　　1、创建用户：elasticsearch

[root@iZbp1bb2egi7w0ueys548pZ bin]# adduser elasticsearch
　　2、创建用户密码，需要输入两次

[root@iZbp1bb2egi7w0ueys548pZ bin]# passwd elasticsearch
　　3、将对应的文件夹权限赋给该用户

[root@iZbp1bb2egi7w0ueys548pZ local]# chown -R elasticsearch elasticsearch-6.0.0
　　4、切换至elasticsearch用户

[root@iZbp1bb2egi7w0ueys548pZ etc]# su elasticsearch 　　5、进入启动目录启动 /usr/local/elasticsearch-6.0.0/bin 使用后台启动方式：./elasticsearch -d

[elasticsearch@vmt10003 bin]$ ./elasticsearch -d
　6、启动后测试 　输入curl ip:9200,如果返回一个json数据说明启动成功
可能遇到的问题
1、启动后访问ip:9200没有显示json

[root@iZbp1bb2egi7w0ueys548qZ ~]# curl 10.132.131.51:9200
curl: (7) Failed connect to 10.132.131.51:9200; Connection refused

　　解决：

　　修改elasticsearch.yml文件，添加

　　network.host: 0.0.0.0

　　再次启动就可以了

[root@iZbp1bb2egi7w0ueys548qZ ~]# curl 10.132.131.51:9200
{
  "name" : "dMD7fZd",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "Cy4a99t2Sw2_hnJ_jtdRgA",
  "version" : {
    "number" : "6.0.0",
    "build_hash" : "8f0685b",
    "build_date" : "2017-11-10T18:41:22.859Z",
    "build_snapshot" : false,
    "lucene_version" : "7.0.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

如果开始了密码功能访问发生应该是这样

 curl --user elastic:abc-123 -XGET '127.0.0.1:9200'

如果某些es的sh程序不能启动的话 还需要设置这些sh的执行权限 chmod -x xxx.sh

没什么问题的话 帆软报表就可以连接ES使用了

elasticsearch密码设置

7.0以后es把基础的安全模块免费集成了。很棒。现在试试设置吧。
官网安全模块文档
其中包含了一个安全入门教程
下面基本都是以我的操作为例，实际上可以按照官网文档进行其它的尝试。
一、设置账号密码
单节点
在elasticsearch.yml文件里增加配置

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
初始化密码需要在es启动的情况下进行设置，按照提示输入各个内置用户的密码。

[esuser@localhost elasticsearch-7.12.0]$./bin/elasticsearch -d
[esuser@localhost elasticsearch-7.12.0]$./bin/elasticsearch-setup-passwords interactive
主要设置了一下的密码

Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

开发问题

springboot中依赖

注意 我这边的es服务端是7.4.2

        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>7.7.0</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>7.7.0</version>
        </dependency>

es没有开启密码验证的配置代码

package com.ag.dtools.config;


import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.transport.client.PreBuiltTransportClient;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.net.InetAddress;

@Configuration
public class ElasticSearchConf {

    private static final Logger LOGGER = LoggerFactory.getLogger(ElasticSearchConf.class);

    /**
     * elk集群地址
     */
    @Value("${elasticsearch.ip}")
    private String hostName;

    /**
     * 端口
     */
    @Value("${elasticsearch.port}")
    private String port;

    /**
     * 集群名称
     */
    @Value("${elasticsearch.cluster.name}")
    private String clusterName;

    /**
     * 连接池
     */
    @Value("${elasticsearch.pool}")
    private String poolSize;

    /**
     * Bean name default  函数名字
     *
     * @return
     */
    @Bean(name = "transportClient")
    public TransportClient transportClient() {
        LOGGER.info("Elasticsearch初始化开始。。。。。");
        TransportClient transportClient = null;
        try {
            // 配置信息
            Settings esSetting = Settings.builder()
                    .put("cluster.name", clusterName) //集群名字
                    .put("client.transport.sniff", true)//增加嗅探机制，找到ES集群
                    .put("thread_pool.search.size", Integer.parseInt(poolSize))//增加线程池个数，暂时设为5
                    .build();
            //配置信息Settings自定义
            transportClient = new PreBuiltTransportClient(esSetting);
            TransportAddress transportAddress = new TransportAddress(InetAddress.getByName(hostName), Integer.valueOf(port));
            transportClient.addTransportAddresses(transportAddress);
        } catch (Exception e) {
            LOGGER.error("elasticsearch TransportClient create error!!", e);
        }
        return transportClient;
    }
}

yml

elasticsearch:
  ip: 112.124.xx.xx
  port: 9200
  pool: 5
  cluster:
    name: my-application

如果es开启了密码登陆需要用这个配置

package com.ag.dtools.config;


import org.apache.http.HttpHost;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.client.ClientConfiguration;
import org.springframework.data.elasticsearch.client.RestClients;
import javax.annotation.PostConstruct;
import java.net.InetAddress;
import java.net.UnknownHostException;


@Configuration
public class ElasticSearchConf {

    private static final Logger LOGGER = LoggerFactory.getLogger(ElasticSearchConf.class);

    /**
     * elk集群地址
     */
    @Value("${elasticsearch.ip}")
    private String hostName;

    /**
     * 端口
     */
    @Value("${elasticsearch.port}")
    private String port;

    /**
     * 集群名称
     */
    @Value("${elasticsearch.cluster.name}")
    private String clusterName;

    /**
     * 连接池
     */
    @Value("${elasticsearch.pool}")
    private String poolSize;
    @Value("${elasticsearch.password}")
    private String password;
    
    /**
     * Bean name default  函数名字
     *
     * @return
     */
    @Bean(name = "restHighLevelClient")
    public RestHighLevelClient transportClient() {
        LOGGER.info("Elasticsearch初始化开始。。。。。");
        final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
                .connectedTo(hostName+":"+port)
                .withBasicAuth("elastic", password)
                .build();
        return RestClients.create(clientConfiguration).rest();
    }
}

同时还需要使用这个配置

elasticsearch:
  ip: 192.168.200.24
  port: 9200
  pool: 5
  username: "elastic"
  password: "es-123456"
  cluster:
    name: my-application

None of the configured nodes are available

这个问题很扯淡,可能各种原因，但我是ES7。。所以没遇到过。。 至于网上说的什么9200 要改成9300的，那是因为是ES6 ES7 直接9200也没有问题

Springboot中消费kafka数据写入ES时报异常

Connection closed unexpectedly 
org.apache.http.ConnectionClosedException: Connection is closed

经过几次测试

发现，项目中使用了nacos 在没有开启nacos服务器的情况下，项目连接nacos 会网络错误，接着 es put 时 就抛出了上面说的异常

nacos 异常

Caused by: com.alibaba.nacos.api.exception.NacosException: failed to req API:/api//nacos/v1/ns/instance after all servers([127.0.0.1:8848]) tried: java.net.ConnectException: Connection refused: connect
	at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:496)
	at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:401)
	at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:397)
	at com.alibaba.nacos.client.naming.net.NamingProxy.registerService(NamingProxy.java:212)
	at com.alibaba.nacos.client.naming.NacosNamingService.registerInstance(NacosNamingService.java:207)
	at com.alibaba.cloud.nacos.registry.NacosServiceRegistry.register(NacosServiceRegistry.java:64)
	... 27 common frames omitted

打开Nacos服务器后，一切正常。
随后关闭nacos，继续消费写入ES，又不报异常了。。。
这个问题 需要继续关注...

网上相关的问题解决方案

https://www.cnblogs.com/quwenli/p/15050215.html
https://www.jianshu.com/p/cccae69f491b

class file for org.elasticsearch.client.Cancellable not found

在新的项目中，按照上面的开发配置是没有问题，但是我在另一个项目中就出现了这个问题，还出现了

java.lang.NoSuchMethodError: org.elasticsearch.action.search.SearchRequest.getPreFilterShardSize()

所以该了下配置

properties 中必须设置ES的版本，不然 迷之报错...

 <properties>
        <java.version>1.8</java.version>
            <elasticsearch.version>7.6.2</elasticsearch.version>
    </properties>

pom

   <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>7.6.2</version>
        </dependency>

        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>7.6.2</version>
        </dependency>

        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
            <version>7.6.2</version>
        </dependency>


        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-elasticsearch</artifactId>
        </dependency>