elk安装搭建

基础环境安装

ELK功能：Logstash用来收集数据，Elasticsearch用来存储数据，Kibana用来展现数据。

实验环境：centos7.5 ,关闭防火墙，关闭selinux 

ip地址：192.168.10.50 部署Kibana、ES

ip地址：192.168.10.51 部署Logstash  

JDK1.8和Kibana安装部署

192.168.10.50

[root@zyxy01 local]# cd /usr/local/src/
[root@zyxy01 src]# tar -zxvf jdk-8u201-linux-x64.tar.gz
[root@zyxy01 src]# mv jdk1.8.0_201 /usr/local/
[root@zyxy01 local]# vim /etc/profile
export JAVA_HOME=/usr/local/jdk1.8.0_201/
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH
[root@zyxy01 local]# source /etc/profile
[root@zyxy01 local]# java -version
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
[root@zyxy01 local]#

　kibana安装启动：

[root@zyxy01 src]# tar -zxvf kibana-6.6.0-linux-x86_64.tar.gz
[root@zyxy01 src]#mv kibana-6.6.0-linux-x86_64 /usr/local/kibana-6.6.0
[root@zyxy01 src]#vim /usr/local/kibana-6.6.0/config/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
#elasticsearch.url: "http://localhost:9200"
#elasticsearch.username: "user"
#elasticsearch.password: "pass"
Kibana的启动和访问：
1、前台启动Kibana：/usr/local/kibana-6.6.0/bin/kibana
2、后台启动Kibana：nohup /usr/local/kibana-6.6.0/bin/kibana >/tmp/kibana.log 2>/tmp/kibana.log &
3、访问Kibana，需要开放5601端口

前端页面出现如下图所示表示安装成功。

 Kibana用Nginx实现认证：

[root@zyxy01 src]#yum install -y lrzsz wget gcc gcc-c++ make pcre pcre-devel zlib zlib-devel
[root@zyxy01 src]#tar -zxvf nginx-1.14.2.tar.gz
[root@zyxy01 src]#cd nginx-1.14.2
[root@zyxy01 src]#./configure --prefix=/usr/local/nginx && make && make install
[root@zyxy01 src]#/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
[root@zyxy01 src]# vim /etc/profile
  export PATH=$PATH:/usr/local/nginx/sbin/
[root@zyxy01 src]# source /etc/profile
[root@zyxy01 src]# nginx -V
#1、nginx限制源ip地址访问，
[root@zyxy01 src]#vim /usr/local/nginx/conf/nginx.conf
 pid        /usr/local/nginx/logs/nginx.pid;  #填写pid路径，打开注释。
#以下四行日志打开注释
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;
#然后配置location限制访问的源ip
 server {
       listen       80;
       location / {
           allow 127.0.0.1;
           allow 192.168.10.1;
           deny all;
           proxy_pass http://127.0.0.1:5601;
       }
    }
#重启nginx生效
[root@zyxy01 nginx-1.14.2]# nginx -s reload
#修改kibana只能本地访问，
[root@zyxy01 src]# vim /usr/local/kibana-6.6.0/config/kibana.yml 
server.host: "127.0.0.1"
#杀kibana进程，重启服务生效。
#Nginx访问日志，可以看到源ip地址192.168.10.1
[root@zyxy01 nginx-1.14.2]# tail -f /usr/local/nginx/logs/access.log 
192.168.10.1 - - [18/May/2020:22:43:50 +0800] "GET / HTTP/1.1" 403 571 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36" "-"
192.168.10.1 - - [18/May/2020:22:43:50 +0800] "GET /favicon.ico HTTP/1.1" 403 571 "http://192.168.10.50/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36" "-"
192.168.10.1 - - [18/May/2020:22:43:53 +0800] "GET / HTTP/1.1" 403 571 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36" "-"
192.168.10.1 - - [18/May/2020:22:43:53 +0800] "GET /favicon.ico HTTP/1.1" 403 571 "http://192.168.10.50/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36" "-"
#2、nginx账号密码登录方式，只需要在1的基础上修改Nginx的 location配置。
location / {
            auth_basic "elk auth";
            auth_basic_user_file /usr/local/nginx/conf/htpasswd;
            proxy_pass http://127.0.0.1:5601;
        }
#重启nginx生效
[root@zyxy01 nginx-1.14.2]# nginx -s reload
#然后设置用户名和密码
[root@zyxy01 nginx-1.14.2]# [root@zyxy01 nginx-1.14.2]# printf "aomo:$(openssl passwd -1 password)
" >/usr/local/nginx/conf/htpasswd
#再次网页访问 192.168.10.50 输入用户名 aomo 密码 password才可以跳转访问到kibana.

　　

 Elasticsearch安装启动操作

[root@zyxy01 ~]# cd /usr/local/src/
[root@zyxy01 src]# tar -zxf elasticsearch-6.6.0.tar.gz 
[root@zyxy01 src]# mv elasticsearch-6.6.0 /usr/local/                    
[root@zyxy01 src]# vim /usr/local/elasticsearch-6.6.0/config/elasticsearch.yml 
#打开以下4行注释并修改存放数据和日志路径，监听端口
path.data: /usr/local/elasticsearch-6.6.0/data
path.logs: /usr/local/elasticsearch-6.6.0/logs
network.host: 0.0.0.0
http.port: 9200           
[root@zyxy01 src]# vim /usr/local/elasticsearch-6.6.0/config/jvm.options 
#JVM的内存限制更改，修改以下两项参数
-Xms128M
-Xmx128M

#Elasticsearch的启动，需要用普通用户启动，root启动会报错。
[root@zyxy01 src]# useradd -s /sbin/nologin elk
[root@zyxy01 src]# chown -R elk:elk /usr/local/elasticsearch-6.6.0/
[root@zyxy01 src]# su - elk -s /bin/bash
#切换到elk用户，启动并查看日志，端口，再次访问kibana页面正常。
[elk@zyxy01 ~]$ /usr/local/elasticsearch-6.6.0/bin/elasticsearch -d
[elk@zyxy01 ~]$ tail -f /usr/local/elasticsearch-6.6.0/logs/elasticsearch.log 
[root@zyxy01 logs]# ss -lnt　　

浏览器输入 192.168.10.50 回车 输入 aomo 密码password出现kibana界面。

Elasticsearch监听在非127.0.0.1 监听在0.0.0.0或者内网地址，以上两种监听都需要调整系统参数。

ES启动监听非127.0.0.1 三个报错的处理需要调整以下三个系统参数： 最大文件打开数、最大打开进程数、内核参数调整。
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: max number of threads [3829] for user [elk] is too low, increase to at least [4096]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

[root@zyxy01 logs]# sysctl -a |grep vm.max_map_count
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.ens33.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
vm.max_map_count = 65530
[root@zyxy01 logs]# vim /etc/sysctl.conf 
[root@zyxy01 logs]# sysctl -p
fs.file-max = 6553560
vm.max_map_count = 262144
[root@zyxy01 logs]# sysctl -a |grep vm.max_map_count
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.ens33.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
vm.max_map_count = 262144
[root@zyxy01 logs]# vim /etc/security/limits.conf
*                soft    nproc           8192

*                hard    nproc           16384

*                soft    nofile          8192

*                hard    nofile          65536
# End of file
[root@zyxy01 logs]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 14989
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 16384
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

　　

Elasticsearch的3个概念:

索引 ->类似于Mysql中的数据库

类型 ->类似于Mysql中的数据表

文档 ->存储数据

Elasticsearch的数据操作常在kibana 左列 Dev Tools里面操作。

创建索引：  PUT /aomo

删除索引：  DELETE /aomo

获取所有索引： GET /_cat/indices?v

Elasticsearch增删改查

ES插入数据
PUT /aomo/users/1
{
  "name":"aomo", 
  "age": 27
}

ES查询数据
11.	GET /aomo/users/1
12.	GET /aomo/_search?q=*

修改数据、覆盖
PUT /aomo/users/1
{
  "name": "it",
  "age": 40
}

ES删除数据
DELETE /aomo/users/1

修改某个字段、不覆盖
POST /aomo/users/2/_update
{
  "doc": {
    "age": 29
  }
}

修改所有的数据
POST /aomo/_update_by_query
{
  "script": {
    "source": "ctx._source['age']=27" 
  },
  "query": {
    "match_all": {}
  }
}

增加一个字段
POST /aomo/_update_by_query
{
  "script":{
    "source": "ctx._source['city']='hangzhou'"
  },
  "query":{
    "match_all": {}
  }
}

Logstash安装和ES结合

192.168.10.51安装logstash，需要安装jdk1.8 然后安装nginx,浏览器访问192.168.10.51触发access.log日志生成，发送到10.50的es,kibana展示日志。

Logstash的安装命令
cd /usr/local/src
tar -zxf logstash-6.6.0.tar.gz
 mv logstash-6.6.0 /usr/local/

Logstash的JVM配置文件更新/usr/local/logstash-6.6.0/config/jvm.options 
-Xms200M
-Xmx200M

Logstash配置发送日志到ES数据库/usr/local/logstash-6.6.0/config/logstash.conf
input {
  file {
    path => "/usr/local/nginx/logs/access.log"
  }
}
output {
  elasticsearch {
    hosts => ["http://192.168.10.50:9200"]
  }
}

Logstash的启动：
yum -y install epel-release
yum install haveged -y
systemctl enable haveged
systemctl start haveged
前台启动：/usr/local/logstash-6.6.0/bin/logstash -f /usr/local/logstash-6.6.0/config/logstash.conf
后台启动：nohup /usr/local/logstash-6.6.0/bin/logstash -f /usr/local/logstash-6.6.0/config/logstash.conf >/tmp/logstash.log 2>/tmp/logstash.log &

　　然后在kibana , Discover--Mangerment 新建索引 logstash*,能看到nginx的访问日志。

 Filebeat的安装

 

 Logstash收集日志：依赖于Java环境，用来收集日志比较重，占用内存和CPU。Filebeat相对轻量，占用服务器资源小，一般选用Filebeat来进行日志收集。

192.168.10.51安装Filebeat

cd /usr/local/src/
tar -zxf filebeat-6.6.0-linux-x86_64.tar.gz
mv filebeat-6.6.0-linux-x86_64 /usr/local/filebeat-6.6.0

Filebeat发送日志到ES配置/usr/local/filebeat-6.6.0/filebeat.yml
filebeat.inputs:
- type: log
  tail_files: true
  backoff: "1s"
  paths:
      - /usr/local/nginx/logs/access.log

output:
  elasticsearch:
    hosts: ["192.168.10.50:9200"]

启动Filebeat
前台启动： /usr/local/filebeat-6.6.0/filebeat  -e -c /usr/local/filebeat-6.6.0/filebeat.yml
后台启动：nohup /usr/local/filebeat-6.6.0/filebeat  -e -c /usr/local/filebeat-6.6.0/filebeat.yml >/tmp/filebeat.log 2>&1 &

Kibana上查看日志数据
GET /xxx/_search?q=*
GET /_cat/indices?v  #查看所有索引

 kibana  Management 创建Filebeat新索引，