背景： 生产环境大量使用 elasticsearch 集群，不同的业务使用不同版本的elasticsearch es经常曝出一些大的漏洞，需要进行版本升级，并且使用x-pack的基本验证功能，避免用户数据泄露 x-pack免费版本特征： 基本的TLS 功能，可对通信进行加密 文件和原生 Realm，可用于创建和管理用户 基于角色的访问控制，可用于控制用户对集群 API 和索引的访问权限； 通过针对 Kibana Spaces 的安全功能，还可允许在 Kibana 中实现多租户。 升级的两种策略： 1.滚动性升级，即不中断业务服务，一台一台进行升级 2.全新部署新版本，然后将数据迁移到新版本的es集群中 这两种方式都需要将数据恢复到新版本的es集群中，可以先进行快照备份 1.升级前先备份低版本的elasticserch的数据：快照方式 原理：即将老版本的es数据打个快照备份出来写入到 /opt/esback 目录中并进行nfs挂载到某台服务器中(作为nfs服务端的机器要有足够量的磁盘空间，最好磁盘性能比较好)，新、旧两个es集群的配置文件中都引用配置 path.repo: ["/opt/esback/"]， 这样新的集群也能对这个目录进行操作了，等待新集群搭建好后，直接把 /opt/esback 目录中的文件恢复到新集群的 索引 index 中即可 使用Mount nfs进行挂载共享(所有的es集群节点都可以访问)： 目标：将本地es备份出来的数据目录/opt/esback 目录挂载到nfs的共享目录 /opt/es_snapshot，这样恢复的时候就都可以访问这个共享目录进行恢复了 // 在10.10.18.92上创建共享目录 创建共享目录，即作为nfs的共享目录 mkdir /opt/es_snapshot 创建本地备份出来的目录 /opt/esback # 在集群所有节点中创建 /opt/esback 目录，即将es数据备份出来的目标目录 # 将其中一台es客户端作为nfs服务端 #nfs服务端的操作 # vim /etc/exports # 注意此处的anonuid和gid要和运行es程序的用户保持一致 # 添加指定 uid 和 gid 的用户 groupadd -g 1000 elastic useradd -u 1000 -g elastic elastic # 修改 gid和 uid为500 命令示例： usermod -u 500 es groupmod -g 500 es /opt/es_snapshot *(insecure,rw,no_root_squash,sync,anonuid=1000,anongid=1000) // 查看共享文件夹 yum install -y exportfs exportfs -rv // nfs服务端修改nfs配置 vim /etc/sysconfig/nfs 修改如下： RPCNFSDARGS="-N 2 -N 3" ----->启用 # Turn off v4 protocol support RPCNFSDARGS="-N 4" ---->启用 重启生效 systemctl restart nfs // 客户端操作 yum install -y nfs-utils // 重启启动新集群机器的NFS服务 systemctl restart nfs //每一台es节点服务器上进行Mount挂载 mount -t nfs 10.10.18.90:/opt/es_snapshot /opt/esback -o proto=tcp -o nolock 列出nfs服务端共享的目录: [root@sz_kp_wanghong_dev02_18_93:/home/wanxing]# showmount -e 10.10.18.90 Export list for 10.10.18.92: /opt/es_snapshot * // 在旧机器上将共享目录的权限付给ES的运行用户 chown elastic:elastic -R /opt/esback 2.创建ES仓库my_backup 修改配置文件： vim elasticsearch.yml # 添加如下配置(需要在旧集群的每个节点上添加)，重新启动集群 path.repo: ["/opt/esback"] 创建快照仓库 my_backup 命令： curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.90:9200/_snapshot/my_backup -d ' { "type": "fs", "settings": { "location": "/opt/esback", "compress": true } } ' # 返回值 {"acknowledged":true} # 报错的处理 'RemoteTransportException[[ictr_node1][10.10.18.93:9300][internal:admin/repository/verify]] # 权限不够 chown -R es.es /opt/es_snapshot/ chown -R es.es /opt/esback_20191104/ # 创建所有索引的备份 # curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.90:9200/_snapshot/my_backup/snapshot20191107 {"accepted":true} 查看备份 [elastic@szyyelk01t slave02]$ curl -XGET http://10.10.18.90:9200/_snapshot/my_backup/snapshot20191107?pretty { "snapshots" : [ { "snapshot" : "snapshot20191107", "uuid" : "0_4SOntVS1GH-7irHjKBMQ", "version_id" : 6030299, "version" : "6.3.2", "indices" : [ "support_faq_categorys", "ticket_list", "templates_search", "site_page_search", "support", "templates_page_search", "support_new_articles", "article_version", "blocks_version", "search", "version", "article_search", "templates", "learn", "templates_version", "blocks_search", "templates_page_version" ], "include_global_state" : true, "state" : "SUCCESS", "start_time" : "2019-11-07T01:35:00.811Z", "start_time_in_millis" : 1573090500811, "end_time" : "2019-11-07T01:35:03.702Z", "end_time_in_millis" : 1573090503702, "duration_in_millis" : 2891, "failures" : [ ], "shards" : { "total" : 71, "failed" : 0, "successful" : 71 } } ] } 升级方式1：滚动升级 elasticsearch5.6.16 --> elasticsearch6.8.4 1.备份数据，避免出现问题后回滚 2.先升级到新版本，然后安装x-pack，此时再要求开发同事修改代码适配 a.先下载新版本的6.8.4 ①关闭自动分片 curl -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{ "persistent": { "cluster.routing.allocation.enable": "none" } }' [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# curl -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{ > "persistent": { > "cluster.routing.allocation.enable": "none" > } > }' * Hostname was NOT found in DNS cache * Trying 10.10.18.92... * Connected to 10.10.18.92 (10.10.18.92) port 9200 (#0) > PUT /_cluster/settings HTTP/1.1 > User-Agent: curl/7.36.0 > Host: 10.10.18.92:9200 > Accept: */* > Content-Length: 73 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 73 out of 73 bytes < HTTP/1.1 200 OK < Warning: 299 Elasticsearch-5.6.15-fe7575a "Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header." "Tue, 05 Nov 2019 08:14:44 GMT" < content-type: application/json; charset=UTF-8 < content-length: 106 < * Connection #0 to host 10.10.18.92 left intact {"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"enable":"none"}}}},"transient":{}} ②暂时禁用非必要的索引并执行同步刷新 curl -XPOST http://10.10.18.92:9200/_flush/synced [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# curl -XPOST http://10.10.18.92:9200/_flush/synced {"_shards":{"total":28,"successful":28,"failed":0},"channel_rel":{"total":4,"successful":4,"failed":0},".kibana":{"total":2,"successful":2,"failed":0},"channel":{"total":6,"successful":6,"failed":0},"video":{"total":4,"successful":4,"failed":0},"channel_list":{"total":6,"successful":6,"failed":0},"influecer":{"total":6,"successful":6,"failed":0}} 注意: 如果是从6.3之前的版本升级上来的，需要注意提前要移除X-Pack插件，然后再去升级版本。执行bin/elasticsearch-plugin remove x-pack a. 备份原来的elasticsearch目录，然后解压新版的elasticsearch。 b. 如果使用外部的配置路径，配置ES_PATH_CONF环境变量到那个位置。如果没有的话，拷贝老的配置目录过来新的elasticsearch目录就可以了。 c. 检查path.data是否指向正确的数据目录 d. 检查path.log是否指向正确的日志目录 新集群的配置文件 [es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ more config/elasticsearch.yml cluster.name: kp-dev-application node.name: ictr_node2 node.master: true node.attr.rack: r1 node.max_local_storage_nodes: 3 network.host: 10.10.18.92 http.port: 9200 transport.tcp.port: 9300 path.repo: ["/opt/esback_20191104"] discovery.zen.minimum_master_nodes: 1 http.cors.enabled: true http.cors.allow-origin: "*" # 新集群的数据还是指向老版本 es5.6.15 的数据存储目录 path.data: /opt/es-node/elasticsearch-5.6.15/data path.logs: /opt/es-node/elasticsearch-5.6.15/logs # 启用安全认证 xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12 ③关闭节点 ④重新启动节点，注意要切换到es用户，不能使用root用户 chown -R es.es elasticsearch-6.8.4 [es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ bin/elasticsearch -d 在其他节点重复以上过程 启动升级后的节点，并通过查看日志和使用下面命令来检查节点是否正确加入到集群 [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl http://10.10.18.92:9200/_cat/nodes 10.10.18.93 16 98 56 1.22 0.50 0.29 di - ictr_node1 10.10.18.92 16 88 8 0.08 0.26 0.31 mdi * ictr_node2 [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl http://10.10.18.92:9200/_cat/indices yellow open channel vRFQoIhmT8WmSbDCfph0ag 3 1 53374 0 44.2mb 44.2mb yellow open channel_rel ZeeBbkogT5KtxzziUYtu_Q 2 1 459528 0 168.8mb 168.8mb yellow open channel_list 1dk8uH8bTeikez0lFR2mJg 3 1 5509390 78630 7gb 7gb yellow open video HNhyt9ioSEayAotGVXRCVg 2 1 798369 228155 1.6gb 1.6gb yellow open .kibana lY82G_-XSniyd_bnMOLuQg 1 1 15 1 146.3kb 146.3kb yellow open influecer RQtQWXKIRE2UYyZlCvv7bA 3 1 148526 48641 272.8mb 272.8mb 节点加入集群后，删除cluster.routing.allocation.enable设置以启用分片分配并开始使用节点： curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{ "persistent": { "cluster.routing.allocation.enable": "all" } }' 重新打开分片报错： [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{ > "persistent": { > "cluster.routing.allocation.enable": "true" > } > }' * Hostname was NOT found in DNS cache * Trying 10.10.18.92... * Connected to 10.10.18.92 (10.10.18.92) port 9200 (#0) > PUT /_cluster/settings HTTP/1.1 > User-Agent: curl/7.36.0 > Host: 10.10.18.92:9200 > Accept: */* > Content-Length: 73 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 73 out of 73 bytes < HTTP/1.1 406 Not Acceptable < content-type: application/json; charset=UTF-8 < content-length: 97 < * Connection #0 to host 10.10.18.92 left intact {"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406} [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl http://10.10.18.92:9200/_cluster/health?pretty { "cluster_name" : "kp-dev-application", "status" : "green", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "active_primary_shards" : 14, "active_shards" : 28, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } 安装新版本中文分词插件 https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.8.4/elasticsearch-analysis-ik-6.8.4.zip # 解压在plugin目录重新启动elasticsearch即可 cd /opt/es-node/elasticsearch-6.8.4/plugins unzip -d elasticsearch-analysis-ik elasticsearch-analysis-ik-6.8.4.zip ot@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4/plugins]# curl http://10.10.18.92:9200/_xpack?pretty { "build" : { "hash" : "bca0c8d", "date" : "2019-10-16T06:19:49.319352Z" }, "license" : { "uid" : "4de9d1c1-59f6-4dfd-8d48-baefd0a583d0", "type" : "basic", "mode" : "basic", "status" : "active" }, "features" : { "ccr" : { "description" : "Cross Cluster Replication", "available" : false, "enabled" : true }, "graph" : { "description" : "Graph Data Exploration for the Elastic Stack", "available" : false, "enabled" : true }, "ilm" : { "description" : "Index lifecycle management for the Elastic Stack", "available" : true, "enabled" : true }, "logstash" : { "description" : "Logstash management component for X-Pack", "available" : false, "enabled" : true }, "ml" : { "description" : "Machine Learning for the Elastic Stack", "available" : false, "enabled" : true, "native_code_info" : { "version" : "6.8.4", "build_hash" : "93ad89b02ff490" } }, "monitoring" : { "description" : "Monitoring for the Elastic Stack", "available" : true, "enabled" : true }, "rollup" : { "description" : "Time series pre-aggregation and rollup", "available" : true, "enabled" : true }, "security" : { "description" : "Security for the Elastic Stack", "available" : true, "enabled" : false }, "sql" : { "description" : "SQL access to Elasticsearch", "available" : true, "enabled" : true }, "watcher" : { "description" : "Alerting, Notification and Automation for the Elastic Stack", "available" : false, "enabled" : true } }, "tagline" : "You know, for X" } 3.启用x-pack的密码验证 # 生成证书 [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# bin/elasticsearch-certutil ca This tool assists you in the generation of X.509 certificates and certificate signing requests for use with SSL/TLS in the Elastic stack. The 'ca' mode generates a new 'certificate authority' This will create a new X.509 certificate and private key that can be used to sign certificate when running in 'cert' mode. Use the 'ca-dn' option if you wish to configure the 'distinguished name' of the certificate authority By default the 'ca' mode produces a single PKCS#12 output file which holds: * The CA certificate * The CA's private key If you elect to generate PEM format certificates (the -pem option), then the output will be a zip file containing individual files for the CA certificate and private key Please enter the desired output file [elastic-stack-ca.p12]: Enter password for elastic-stack-ca.p12 : [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# ls bin config elastic-stack-ca.p12 lib LICENSE.txt logs modules NOTICE.txt plugins README.textile [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 This tool assists you in the generation of X.509 certificates and certificate signing requests for use with SSL/TLS in the Elastic stack. The 'cert' mode generates X.509 certificate and private keys. * By default, this generates a single certificate and key for use on a single instance. * The '-multiple' option will prompt you to enter details for multiple instances and will generate a certificate and key for each one * The '-in' option allows for the certificate generation to be automated by describing the details of each instance in a YAML file * An instance is any piece of the Elastic Stack that requires an SSL certificate. Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats may all require a certificate and private key. * The minimum required value for each instance is a name. This can simply be the hostname, which will be used as the Common Name of the certificate. A full distinguished name may also be used. * A filename value may be required for each instance. This is necessary when the name would result in an invalid file or directory name. The name provided here is used as the directory name (within the zip) and the prefix for the key and certificate files. The filename is required if you are prompted and the name is not displayed in the prompt. * IP addresses and DNS names are optional. Multiple values can be specified as a comma separated string. If no IP addresses or DNS names are provided, you may disable hostname verification in your SSL configuration. * All certificates generated by this tool will be signed by a certificate authority (CA). * The tool can automatically generate a new CA for you, or you can provide your own with the -ca or -ca-cert command line options. By default the 'cert' mode produces a single PKCS#12 output file which holds: * The instance certificate * The private key for the instance certificate * The CA certificate If you specify any of the following options: * -pem (PEM formatted output) * -keep-ca-key (retain generated CA key) * -multiple (generate multiple certificates) * -in (generate certificates from an input file) then the output will be be a zip file containing individual certificate/key files Enter password for CA (elastic-stack-ca.p12) : Please enter the desired output file [elastic-certificates.p12]: Enter password for elastic-certificates.p12 : Certificates written to /opt/es-node/elasticsearch-6.8.4/elastic-certificates.p12 This file should be properly secured as it contains the private key for your instance. This file is a self contained file and can be copied and used 'as is' For each Elastic product that you wish to configure, you should copy this '.p12' file to the relevant configuration directory and then follow the SSL configuration instructions in the product guide. For client applications, you may only need to copy the CA certificate and configure the client to trust this certificate. # 修改config/elasticsearch.yml配置 xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /usr/local/elasticsearch/config/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /usr/local/elasticsearch/config/elastic-certificates.p12 # 配置密码 [es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ bin/elasticsearch-setup-passwords interactive Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user. You will be prompted to enter passwords as the process progresses. Please confirm that you would like to continue [y/N]y Enter password for [elastic]: Reenter password for [elastic]: Enter password for [apm_system]: Reenter password for [apm_system]: Enter password for [kibana]: Reenter password for [kibana]: Enter password for [logstash_system]: Reenter password for [logstash_system]: Enter password for [beats_system]: Reenter password for [beats_system]: Enter password for [remote_monitoring_user]: Reenter password for [remote_monitoring_user]: Changed password for user [apm_system] Changed password for user [kibana] Changed password for user [logstash_system] Changed password for user [beats_system] Changed password for user [remote_monitoring_user] Changed password for user [elastic] 密码：espass [es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ curl --user elastic:espass -XGET 'http://10.10.18.92:9200/_cat/indices' green open channel_rel ZeeBbkogT5KtxzziUYtu_Q 2 1 459528 0 337.7mb 168.8mb green open .security-6 iQHndFBqRe2Ss2o7KMxyFg 1 1 6 0 38.3kb 19.1kb green open .kibana lY82G_-XSniyd_bnMOLuQg 1 1 15 1 292.6kb 146.3kb green open influecer RQtQWXKIRE2UYyZlCvv7bA 3 1 148526 48641 545.6mb 272.8mb green open channel vRFQoIhmT8WmSbDCfph0ag 3 1 53374 0 88.4mb 44.2mb green open channel_list 1dk8uH8bTeikez0lFR2mJg 3 1 5522172 78630 14gb 7gb green open video HNhyt9ioSEayAotGVXRCVg 2 1 798369 228155 3.3gb 1.6gb 升级方式2：完全重启集群升级 即配置好全新的elasticsearch7.4.2集群，然后把数据恢复到新集群中 下载地址：wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.2-linux-x86_64.tar.gz 老版本的配置 # cms elasticsearch [root@szyyelk01t opt]# egrep -v '^#|^$' elk-master/config/elasticsearch.yml cluster.name: cms-uat-elastic node.name: master path.data: /opt/elk-master/data/data01,/opt/elk-master/data/data02 path.logs: /opt/elk-master/logs bootstrap.memory_lock: false bootstrap.system_call_filter: false network.host: 10.10.18.90 http.port: 9200 http.cors.enabled: true http.cors.allow-origin: "*" [root@szyyelk01t elk-slave]# egrep -v '^$|^#' slave01/config/elasticsearch.yml cluster.name: cms-uat-elastic node.name: slave01 path.data: /opt/elk-slave/slave01/data/data01,/opt/elk-slave/slave01/data/data02 path.logs: /opt/elk-slave/slave01/logs bootstrap.memory_lock: false bootstrap.system_call_filter: false network.host: 10.10.18.90 http.port: 8200 discovery.zen.ping.unicast.hosts: ["10.10.18.90"] http.cors.enabled: true http.cors.allow-origin: "*" [root@szyyelk01t elk-slave]# egrep -v '^$|^#' slave02/config/elasticsearch.yml cluster.name: cms-uat-elastic node.name: slave02 path.data: /opt/elk-slave/slave02/data/data01,/opt/elk-slave/slave02/data/data02 path.logs: /opt/elk-slave/slave02/logs bootstrap.memory_lock: false bootstrap.system_call_filter: false network.host: 10.10.18.90 http.port: 8201 discovery.zen.ping.unicast.hosts: ["10.10.18.90"] http.cors.enabled: true http.cors.allow-origin: "*" # 已经升级的线上配置参考： [root@eus_filmora_db01:/usr/local/elasticsearch-7.4.1]# egrep -v '^$|^#' config/elasticsearch.yml cluster.name: UOS_CLUSTER_ES node.name: uos_node_1 path.data: /data/elasticsearch_data/data path.logs: /data/elasticsearch_data/logs bootstrap.memory_lock: true network.host: 172.20.103.199 http.port: 9200 transport.tcp.port: 9300 node.master: true node.data: true discovery.seed_hosts: ["172.20.103.199:9300", "172.20.73.200:9300", "172.20.73.212:9300"] cluster.initial_master_nodes: ["172.20.103.199", "172.20.73.200", "172.20.73.212"] gateway.recover_after_nodes: 2 transport.tcp.compress: true path.repo: ["/data/bak_es"] xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /usr/local/elasticsearch/config/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /usr/local/elasticsearch/config/elastic-certificates.p12 ############## cms 系统的 elasticsearch 6升级7.4.2 整体策略： 1.升级cms测试环境的es到7.4.2，然后做适配性的开发，再次升级内网的生产环境 --> cms海外环境(找海外业务不繁忙的时候操作，先和开发协商好) 测试环境其他人依赖进行测试，所以升级需要两套并存，新版本的es7.4.2使用自带的openjdk13.0 1.配置新版本的elasticsearch使用指定的jdk环境 # vim bin/elasticsear export JAVA_HOME=/opt/elk7_onenode/elasticsearch-7.4.2/jdk export PATH=$JAVA_HOME/bin:$PATH mkdir /opt/elk7_onenode/elasticsearch-7.4.2/data # 主节点配置 [elastic@szyyelk01t elasticsearch-7.4.2]$ more config/elasticsearch.yml cluster.name: cms-uat-elastic7 node.name: cms_node01 node.master: true node.data: true discovery.seed_hosts: ["10.10.18.90:19300", "10.10.18.117:19300"] cluster.initial_master_nodes: ["10.10.18.90"] path.data: /opt/cms_elk7/elasticsearch-7.4.2/data path.logs: /opt/cms_elk7/elasticsearch-7.4.2/logs discovery.zen.minimum_master_nodes: 1 bootstrap.memory_lock: false bootstrap.system_call_filter: false network.host: 10.10.18.90 http.cors.enabled: true http.cors.allow-origin: "*" transport.tcp.compress: true path.repo: ["/opt/esback/"] gateway.recover_after_nodes: 1 # 增加新的参数head插件可以访问es http.port: 19200 transport.tcp.port: 19300 gateway.recover_after_time: 8m # 以下配置可以减少当es节点短时间宕机或重启时shards重新分布带来的磁盘io读写浪费 discovery.zen.fd.ping_timeout: 300s discovery.zen.fd.ping_retries: 8 discovery.zen.fd.ping_interval: 30s discovery.zen.ping_timeout: 180s # 启用安全认证 xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12 # 第二个节点配置 [elastic@cms-test:/opt/cms_elk7/elasticsearch-7.4.2]$ more config/elasticsearch.yml cluster.name: cms-uat-elastic7 node.name: cms_node02 node.master: false node.data: true discovery.seed_hosts: ["10.10.18.90:19300", "10.10.18.117:19300"] cluster.initial_master_nodes: ["10.10.18.90"] path.data: /opt/cms_elk7/elasticsearch-7.4.2/data path.logs: /opt/cms_elk7/elasticsearch-7.4.2/logs bootstrap.memory_lock: false bootstrap.system_call_filter: false network.host: 10.10.18.117 http.cors.enabled: true http.cors.allow-origin: "*" transport.tcp.compress: true path.repo: ["/opt/esback/"] gateway.recover_after_nodes: 1 # 增加新的参数head插件可以访问es http.port: 19200 transport.tcp.port: 19300 gateway.recover_after_time: 8m # 以下配置可以减少当es节点短时间宕机或重启时shards重新分布带来的磁盘io读写浪费 discovery.zen.fd.ping_timeout: 300s discovery.zen.fd.ping_retries: 8 discovery.zen.fd.ping_interval: 30s discovery.zen.ping_timeout: 180s # 启用安全认证 xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12 # 设置密码 # 启用安全认证，只添加这个选项 xpack.security.enabled: true #xpack.security.transport.ssl.enabled: true #xpack.security.transport.ssl.verification_mode: certificate #xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 #xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12 elastic密码： espass 在集群上配置TLS: 如果你在操作单节点ES则可以跳过本内容。 1.生成CA证书 : bin/elasticsearch-certutil ca 将产生新文件 elastic-stack-ca.p12。该 elasticsearch-certutil 命令还会提示你输入密码以保护文件和密钥，请保留该文件的副本并记住其密码，此处我们设置为空 2.为集群中的每个节点生成证书和私钥 bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 将产生新文件 elastic-certificates.p12。系统还会提示你输入密码，你可以输入证书和密钥的密码，也可以按Enter键将密码留空。默认情况下 elasticsearch-certutil 生成没有主机名信息的证书，这意味着你可以将证书用于集群中的每个节点，另外要关闭主机名验证。 将 elastic-certificates.p12 文件复制到每个节点上Elasticsearch配置目录中 无需将 elastic-stack-ca.p12 文件复制到此目录。 mkdir config/certs mv elastic-certificates.p12 config/certs/ 配置集群中的每个节点以使用其签名证书标识自身并在传输层上启用TLS 启用TLS并指定访问节点证书所需的信息，将以下信息添加到每个节点的 elasticsearch.yml 文件中： xpack.security.enabled: true 3.设置密码 # 报错 [elastic@szyyelk01t elasticsearch-7.4.2]$ bin/elasticsearch-setup-passwords interactive Failed to determine the health of the cluster running at http://10.10.18.90:19200 Unexpected response code [503] from calling GET http://10.10.18.90:19200/_cluster/health?pretty Cause: master_not_discovered_exception It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords. It is very likely that the password changes will fail when run against an unhealthy cluster. Do you want to continue with the password setup process [y/N]y Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user. You will be prompted to enter passwords as the process progresses. Please confirm that you would like to continue [y/N]y Enter password for [elastic]: Reenter password for [elastic]: Enter password for [apm_system]: Reenter password for [apm_system]: Enter password for [kibana]: Reenter password for [kibana]: Enter password for [logstash_system]: Reenter password for [logstash_system]: Enter password for [beats_system]: Reenter password for [beats_system]: Enter password for [remote_monitoring_user]: Reenter password for [remote_monitoring_user]: Unexpected response code [503] from calling PUT http://10.10.18.90:19200/_security/user/apm_system/_password?pretty Cause: Cluster state has not been recovered yet, cannot write to the [null] index Possible next steps: * Try running this tool again. * Try running with the --verbose parameter for additional messages. * Check the elasticsearch logs for additional error details. * Use the change password API manually. ERROR: Failed to set password for user [apm_system]. [elastic@szyyelk01t elasticsearch-7.4.2]$ bin/elasticsearch-setup-passwords interactive Connection failure to: http://10.10.18.90:19200/_security/_authenticate?pretty failed: Connection refused ERROR: Failed to connect to elasticsearch at http://10.10.18.90:19200/_security/_authenticate?pretty. Is the URL correct and elasticsearch running? [elastic@szyyelk01t elasticsearch-7.4.2]$ bin/elasticsearch-setup-passwords interactive Failed to determine the health of the cluster running at http://10.10.18.90:19200 Unexpected response code [503] from calling GET http://10.10.18.90:19200/_cluster/health?pretty Cause: master_not_discovered_exception It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords. It is very likely that the password changes will fail when run against an unhealthy cluster. Do you want to continue with the password setup process [y/N]^C[elastic@szyyelk01t elasticsearch-7.4.2]$ bin/elasticsearch-setup-passwords interactive Failed to determine the health of the cluster running at http://10.10.18.90:19200 Unexpected response code [503] from calling GET http://10.10.18.90:19200/_cluster/health?pretty Cause: master_not_discovered_exception It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords. It is very likely that the password changes will fail when run against an unhealthy cluster. Do you want to continue with the password setup process [y/N] 解决办法：只配置一个主节点：cluster.initial_master_nodes: ["10.10.18.90"] # 错误2处理 [2019-11-07T16:12:31,563][INFO ][o.e.c.c.JoinHelper ] [cms_node02] failed to join {cms_node01}{765pAegcS8S0Y3OrE9taMA}{Up16Gw9pQlyXg3n1wCHE8g}{10.10.18.90}{10.10.18.90:19300}{dilm}{ml.machine_memory=8362151936, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={cms_node02}{765pAegcS8S0Y3OrE9taMA}{ki1VVW27TnakEEFagCoDlg}{10.10.18.117}{10.10.18.117:19300}{dil}{ml.machine_memory=16853446656, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={cms_node02}{765pAegcS8S0Y3OrE9taMA}{ki1VVW27TnakEEFagCoDlg}{10.10.18.117}{10.10.18.117:19300}{dil}{ml.machine_memory=16853446656, xpack.installed=true, ml.max_open_jobs=20}, targetNode={cms_node01}{765pAegcS8S0Y3OrE9taMA}{Up16Gw9pQlyXg3n1wCHE8g}{10.10.18.90}{10.10.18.90:19300}{dilm}{ml.machine_memory=8362151936, ml.max_open_jobs=20, xpack.installed=true}}]} org.elasticsearch.transport.RemoteTransportException: [cms_node01][10.10.18.90:19300][internal:cluster/coordination/join] Caused by: java.lang.IllegalArgumentException: can't add node {cms_node02}{765pAegcS8S0Y3OrE9taMA}{ki1VVW27TnakEEFagCoDlg}{10.10.18.117}{10.10.18.117:19300}{dil}{ml.machine_memory=16853446656, ml.max_open_jobs=20, xpack.installed=true}, found existing node {cms_node01}{765pAegcS8S0Y3OrE9taMA}{Up16Gw9pQlyXg3n1wCHE8g}{10.10.18.90}{10.10.18.90:19300}{dilm}{ml.machine_memory=8362151936, xpack.installed=true, ml.max_open_jobs=20} with the same id but is a different node instance at org.elasticsearch.cluster.node.DiscoveryNodes$Builder.add(DiscoveryNodes.java:618) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.coordination.JoinTaskExecutor.execute(JoinTaskExecutor.java:147) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.coordination.JoinHelper$1.execute(JoinHelper.java:119) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-7.4.2.jar:7.4.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] # 拷贝连着目录 都拷贝，删除 data 下面的所有文件重启即可 # 最终密码配置成功 [elastic@szyyelk01t elasticsearch-7.4.2]$ bin/elasticsearch-setup-passwords interactive Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user. You will be prompted to enter passwords as the process progresses. Please confirm that you would like to continue [y/N]y Enter password for [elastic]: Reenter password for [elastic]: Enter password for [apm_system]: Reenter password for [apm_system]: Enter password for [kibana]: Reenter password for [kibana]: Enter password for [logstash_system]: Reenter password for [logstash_system]: Enter password for [beats_system]: Reenter password for [beats_system]: Enter password for [remote_monitoring_user]: Reenter password for [remote_monitoring_user]: Changed password for user [apm_system] Changed password for user [kibana] Changed password for user [logstash_system] Changed password for user [beats_system] Changed password for user [remote_monitoring_user] Changed password for user [elastic] # 查看集群状态 [elastic@szyyelk01t elasticsearch-7.4.2]$ curl -H "Content-Type: application/json" -u elastic:espass http://10.10.18.90:19200/_cluster/health?pretty { "cluster_name" : "cms-uat-elastic7", "status" : "green", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "active_primary_shards" : 1, "active_shards" : 2, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } # 查看刚才创建所有索引的备份 # curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.90:9200/_snapshot/my_backup/snapshot20191107 {"accepted":true} # 恢复全索引快照 #保证elasticsearch用户拥有快照目录的权限 chown -R elastic.elastic /opt/esback #创建仓库 curl -H "Content-Type: application/json" -XPUT -u elastic:espass http://10.10.18.90:19200/_snapshot/backup -d ' { "type":"fs", "settings":{"location":"/opt/esback"} }' #查询全索引快照备份 $ curl -XGET -u elastic:espass "http://10.10.18.90:19200/_snapshot/backup/_all" | python -m json.tool % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 700 100 700 0 0 160k 0 --:--:-- --:--:-- --:--:-- 170k { "snapshots": [ { "duration_in_millis": 2891, "end_time": "2019-11-07T01:35:03.702Z", "end_time_in_millis": 1573090503702, "failures": [], "include_global_state": true, "indices": [ "support_faq_categorys", "ticket_list", "templates_search", "site_page_search", "support", "templates_page_search", "support_new_articles", "article_version", "blocks_version", "search", "version", "article_search", "templates", "learn", "templates_version", "blocks_search", "templates_page_version" ], "shards": { "failed": 0, "successful": 71, "total": 71 }, "snapshot": "snapshot20191107", "start_time": "2019-11-07T01:35:00.811Z", "start_time_in_millis": 1573090500811, "state": "SUCCESS", "uuid": "0_4SOntVS1GH-7irHjKBMQ", "version": "6.3.2", "version_id": 6030299 } ] } #恢复全索引快照 [elastic@szyyelk01t elasticsearch-7.4.2]$ curl -XPOST -u elastic:espass 'http://10.10.18.90:19200/_snapshot/backup/snapshot20191107/_restore'?wait_for_completion=true {"snapshot":{"snapshot":"snapshot20191107","indices":["templates_page_search","article_search","blocks_version","learn","templates_page_version","templates","version","site_page_search","support_new_articles","support_faq_categorys","search","templates_search","blocks_search","ticket_list","article_version","support","templates_version"],"shards":{"total":71,"failed":0,"successful":71}}} # 查看已经恢复成功 [elastic@szyyelk01t elasticsearch-7.4.2]$ curl -H "Content-Type: application/json" -u elastic:espass http://10.10.18.90:19200/_cat/indices green open templates_page_search tUKh1vaHRla6QamphIByLQ 5 1 104 10 965.3kb 482.6kb green open article_search _LE5n_-KRSGVH6Z3I1YLNQ 5 1 44 2 1.5mb 797.8kb green open blocks_version VRmv8fyESY6iclBYkhKJ_w 5 1 9 0 145.5kb 72.7kb green open learn W4RyJnkrStaRJwQgS4MAug 3 1 89 1 841.6kb 420.8kb green open templates_page_version _hHckKOfRuCPEojviySxVw 5 1 945 0 1.5mb 777kb green open templates 7iJqDoBwTbOEHcyEzPLHbA 5 1 138 0 2mb 1mb green open version mLbfHoA7SAu4RWHSHM3vtw 3 1 1 0 39.9kb 19.9kb green open support_new_articles HvGe-CklRU-iua-_T1pLNA 3 1 1534 170 12mb 6mb green open site_page_search xxk8IetTSr2HF2tEe2Vc1w 5 1 516 2 1.5mb 817.2kb green open .security-7 xdRnCeykQGGPcqM3-_WFCw 1 1 6 0 39.5kb 19.8kb green open search fOteaZd0QfaU_2fKBaWPdA 3 1 0 0 1.5kb 783b green open support_faq_categorys h61nZp5bSQqV1UGVyHL7WA 3 1 0 0 1.5kb 783b green open templates_search ru8oFeQDTtKovOmkjP6A0w 5 1 111 3 1.5mb 802.8kb green open blocks_search 8vMOY6ebTs-4iJIwM2VG0Q 5 1 0 0 2.5kb 1.2kb green open article_version qcF3Nft6QMezKqtPHyYLlA 5 1 344 0 5mb 2.5mb green open ticket_list xpvXuhlqRFq5Y_zugq0qKw 3 1 403 0 2.1mb 1mb green open support LypmJq0pRDy428-TKOy6Yg 3 1 0 0 1.5kb 783b green open templates_version gI28sYWJT3GVgfBeyJhSLg 5 1 220 0 4.2mb 2.1mb