一、安装elasticsearch
官网：https://www.elastic.co/guide/index.html
https://www.elastic.co/guide/en/elasticsearch/reference/2.3/index.html
参考搭建文档
Elasticsearch-6.0.0 logstash-6.0.0 kibana-6.0.0 filebeat-6.0.0 https://blog.51cto.com/zero01/2079879 上篇 https://blog.51cto.com/zero01/2082794 下篇 参考 https://www.cnblogs.com/superlinux/p/10591428.html
filebeat写入kafka :https://www.jianshu.com/p/da8113f58115
1）环境准备
[root@k8s6 ~]# java -version openjdk version "1.8.0_201" OpenJDK Runtime Environment (build 1.8.0_201-b09) OpenJDK 64-Bit Server VM (build 25.201-b09, mixed mode) 1) [root@k8s6 ~]# ls elktools/ elasticsearch-2.3.5.rpm kibana-4.5.4-1.x86_64.rpm logstash-2.3.4-1.noarch.rpm
2）rpm包安装
[root@k8s6 ~]# rpm -ivh elktools/elasticsearch-2.3.5.rpm
3）修改配置文件
备份配置文件 cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak 修改配置文件 [root@k8s6 elasticsearch]# diff /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak 17c17 < cluster.name: myes --- > # cluster.name: my-application 23c23 < node.name: master_node --- > # node.name: node-1 54c54 < network.host: 192.168.10.22 --- > # network.host: 192.168.0.1 58c58 < http.port: 9200 --- > # http.port: 9200
补充：
bootstrap.mlockall: false bootstrap.system_call_filter: false
4）启动服务
[root@k8s6 elasticsearch]# systemctl start elasticsearch 启动服务 [root@k8s6 elasticsearch]# netstat -lntup|grep java tcp6 0 0 192.168.10.22:9200 :::* LISTEN 25874/java tcp6 0 0 192.168.10.22:9300 :::* LISTEN 25874/java
5）对elasticsearch的api验证
[root@k8s6 elasticsearch]# curl http://192.168.10.22:9200 { "name" : "master_node", "cluster_name" : "myes", "version" : { "number" : "2.3.5", "build_hash" : "90f439ff60a3c0f497f91663701e64ccd01edbb4", "build_timestamp" : "2016-07-27T10:36:52Z", "build_snapshot" : false, "lucene_version" : "5.5.0" }, "tagline" : "You Know, for Search" }
二、安装elasticsearch的插件安装
1）查看执行的命令和插件位置
[root@k8s6 ~]# ls /usr/share/elasticsearch/
bin lib LICENSE.txt modules NOTICE.txt plugins README.textile
2）安装head插件
[root@k8s6 ~]# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head 从github下载 [root@k8s6 ~]# ls /usr/share/elasticsearch/plugins/head/ Dockerfile Gruntfile.js LICENCE proxy src Dockerfile-alpine grunt_fileSets.js package.json README.textile test elasticsearch-head.sublime-project index.html plugin-descriptor.properties _site
2.1）访问测试
http://192.168.10.22:9200/_plugin/head/
3.1）安装 kopf 插件
[root@k8s6 ~]# /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf
3.2）访问测试
http://192.168.10.22:9200/_plugin/kopf/
4）模拟提交post请求
刷新连接
二、es的集群测试
1）另一台机器同样安装好es,修改配置文件（主播模式）测试vm会存在问题
[root@node01 elasticsearch]# grep '^[a-Z]' elasticsearch.yml cluster.name: myes node.name: node01 path.data: /data/es-data path.logs: /var/log/elasticsearch bootstrap.mlockall: false bootstrap.system_call_filter: false network.host: 192.168.10.23 http.port: 9200
创建目录，并授权
[root@node01 elasticsearch]# mkdir -p /data/es-data
[root@node01 elasticsearch]# chown elasticsearch:elasticsearch /data/es-data/
2）单播模式
[root@node01 elasticsearch]# grep '^[a-Z]' elasticsearch.yml cluster.name: myes node.name: node01 path.data: /data/es-data path.logs: /var/log/elasticsearch bootstrap.mlockall: true network.host: 192.168.10.23 http.port: 9200 discovery.zen.ping.unicast.hosts: ["192.168.10.22", "192.168.10.23"]
3）启动服务
强调，一点要关闭防火墙。否则无法弄成集群
[root@node01 elasticsearch]# /etc/init.d/elasticsearch start
4）查询集群的状态
[root@node01 ~]# curl -XGET 'http://192.168.10.22:9200/_cluster/health?pretty=true' { "cluster_name" : "myes", "status" : "green", "timed_out" : false, "number_of_nodes" : 2, # 2个节点 "number_of_data_nodes" : 2, "active_primary_shards" : 5, "active_shards" : 10, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }
三、在主节点安装 logstash
1）安装logstash
[root@k8s6 elktools]# rpm -ivh logstash-2.3.4-1.noarch.rpm 准备中... ################################# [100%] 正在升级/安装... 1:logstash-1:2.3.4-1 ################################# [100%] [root@k8s6 elktools]# rpm -qa|grep logstash logstash-2.3.4-1.noarch [root@k8s6 elktools]# rpm -ql logstash # 查看安装生成了哪些文件
2）模拟日志标准输出：/opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'
[root@k8s6 elktools]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }' Settings: Default pipeline workers: 4 Pipeline main started hello 2019-03-15T04:42:43.106Z k8s6 hello hello world 2019-03-15T04:42:51.121Z k8s6 hello world
json格式输出： /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug } }'
[root@k8s6 elktools]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug } }' Settings: Default pipeline workers: 4 Pipeline main started hello world { "message" => "hello world", "@version" => "1", "@timestamp" => "2019-03-15T04:47:12.509Z", "host" => "k8s6" }
3）命令行模拟输出到elasticsearch： /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch{ hosts => ["192.168.10.22:9200"] index => "logstash-%{+YYYY.MM.dd}" } }'
[root@k8s6 elktools]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch{ hosts => ["192.168.10.22:9200"] index => "logstash-%{+YYYY.MM.dd}" } }' Settings: Default pipeline workers: 4 Pipeline main started hhhee
查询到输出的值
4）同时进行屏幕输出并写入es
[root@k8s6 elktools]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug } elasticsearch{ hosts => ["192.168.10.22:9200"] index => "logstash-%{+YYYY.MM.dd}" } }' Settings: Default pipeline workers: 4 Pipeline main started 天天向上 { "message" => "天天向上", "@version" => "1", "@timestamp" => "2019-03-15T05:02:55.388Z", "host" => "k8s6" }
四、使用配置文件启动logstash
检测语法
[root@node01 ~]# /opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/nginx.conf
Configuration OK
1）进入到配置文件目录。cd /etc/logstash/conf.d/ ，该目录由/etc/init.d/logstash 这里定义的
[root@k8s6 elktools]# cd /etc/logstash/conf.d/
[root@k8s6 conf.d]# ls
[root@k8s6 conf.d]#
编辑 demo.conf文件，既输出在屏幕，也存入es
YYYY.MM.dd 每天
[root@k8s6 conf.d]# cat demo.conf input { stdin{} } filter{ } output{ elasticsearch { hosts => ["192.168.10.22:9200"] index => "logstash-%{+YYYY.MM.dd}" } stdout{ codec => rubydebug } }
启动服务：[root@k8s6 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/demo.conf
2)使用file插件，指定要记录日志的目录
[root@k8s6 conf.d]# cat file.conf input { file{ path => ["/var/log/messages", "/var/log/secure"] type => "system-log" start_position => "beginning" } } filter{ } output{ elasticsearch { hosts => ["192.168.10.22:9200"] index => "system-log-%{+YYYY.MM}" } }
启动服务： /opt/logstash/bin/logstash -f /etc/logstash/conf.d/file.conf
五、安装kibana
1）在主节点安装kibana
[root@k8s6 elktools]# rpm -ivh kibana-4.5.4-1.x86_64.rpm
查看安装到了哪些位置
[root@k8s6 elktools]# rpm -ql kibana
2）修改配置文件
[root@k8s6 config]# pwd /opt/kibana/config [root@k8s6 config]# grep '^[a-Z]' kibana.yml server.port: 5601 server.host: "0.0.0.0" elasticsearch.url: "http://192.168.10.22:9200" kibana.index: ".kibana"
启动服务：[root@k8s6 config]# /etc/init.d/kibana start
[root@k8s6 config]# netstat -lntup|grep 5601 tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 31271/node
网页访问：192.168.10.22:5601
2）配置索引，配置哪个，显示哪个的日志文件
匹配到了创建，即可
补充知识
import requests from requests.auth import HTTPBasicAuth res = requests.get('https://elk0001.eniot.io:9200/_cluster/health/?pretty',verify=False, auth=HTTPBasicAuth('user', 'password')) print(res.text) # curl --insecure --user user:password -XGET "https://elk-log0001.eniot.io:9200/_cluster/health/?pretty"