ELK7.4.0单节点部署
环境准备
-
安装系统,数据盘设置为/srv
-
内核优化参考
-
我们需要创建elk专用的账号,并创建所需要的目录并授权
useradd elk;
mkdir /srv/{app,data,logs}/elk
chown -Rf elk:elk /srv/{app,data,logs}/elk
- 修改
/etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
* soft nproc 65536
* hard nproc 65536
elk soft nofile 65536
elk hard nofile 65536
elk soft nproc 65536
elk hard nproc 65536
安装elk过程所有操作必须使用elk账户进行！
su - elk
Elasticsearch
这次先用的是单节点的ES,没有部署集群,集群部署的后续会更新
首先我们需要从官网下载最新的es安装包,这里建议使用tar包安装;
cd /srv/app/elk;
wget http://172.19.30.116/mirror/elk/elasticsearch/7.4.0/elasticsearch-7.4.0-linux-x86_64.tar.gz
tar -zxvf elasticsearch-7.4.0-linux-x86_64.tar.gz
mv elasticsearch-7.4.0-linux-x86_64.tar.gz elasticsearch
- 修改es的配置文件
/srv/app/elk/elasticsearch/config/elasticsearch.yml
cluster.name: es-cluster
node.name: es-1
node.master: true #允许为master节点
node.data: true #允许为数据节点
path.data: /srv/data/elk/elasticsearch #设置数据目录
path.logs: /srv/logs/elk/elasticsearch #设置日志目录
network.host: 127.0.0.1 #仅允许本地访问,如要其它网段访问,可以设置为网段地址,也可以直接写成0.0.0.0
http.port: 9200 #http端口,默认为9200
http.cors.enabled: true
http.cors.allow-origin: "*"
xpack.security.enabled: false
- JVM内存我给了4G,查询的量也比较小,小一点没关系,根据使用情况来看,es7.4.0版本比较耗费内存
-Xms4g
-Xmx4g
- 启动es, 生产环境不建议如下这种方式启动,建议使用supervisord服务启动,建议参考: https://www.cnblogs.com/lizhaojun-ops/p/11962485.html
/srv/app/elk/elasticsearch/bin/elasticsearch -d
Kibana
cd /srv/app/elk;
wget http://172.19.30.116/mirror/elk/kibana/7.4.0/kibana-7.4.0-linux-x86_64.tar.gz
tar -zxvf kibana-7.4.0-linux-x86_64.tar.gz
mv kibana-7.4.0-linux-x86_64 kibana
- 修改kafka的配置文件 ``
server.port: 5601
server.host: "localhost" #也可以直接写成0.0.0.0
server.name: "kibana"
elasticsearch.hosts: ["http://127.0.0.1:9200"]
i18n.locale: "en" #如果要开启中文可以改成zh-CN
- 启动kibana,生产环境不建议如下这种方式启动,建议使用supervisord服务启动,建议参考: https://www.cnblogs.com/lizhaojun-ops/p/11962485.html
/srv/app/elk/kibana/bin/kibana
logstash
cd /srv/app/elk;
wget http://172.19.30.116/mirror/elk/logstash/7.4.0/logstash-7.4.0.tar.gz
tar -zxvf logstash-7.4.0.tar.gz
mv logstash-7.4.0 logstash
- 根据实际情况修改jvm
/srv/app/elk/logstash/config/jvm.options; 默认1G,如果日志的数量比较大,可以改成2G或者更多
-Xms1g
-Xmx1g
至此,ELK集群已经部署完成了,现在我们需要准备我们的Redis和filebeat了,redis用来做日志的暂存队列,filebeat收集nginx或者其他应用的日志
REDIS
yum install epel-release -y
yum install redis* -y
chkconfig redis on
service redis start
Filebeat
在nginx节点上安装filebeat，修改nginx的log_format,新增nginxjson,并让日志引用这个格式的日志，可以参考这篇博客：
log_format nginxjson '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"service":"nginx",'
'"trace":"$upstream_http_ctx_transaction_id",'
'"clientip":"$remote_addr",'
'"remote_user":"$remote_user",'
'"request":"$request",'
'"url":"$scheme://$http_host$request_uri",'
'"http_user_agent":"$http_user_agent",'
'"server_protocol":"$server_protocol",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"x_clientOs":"$http_x_clientOs",'
'"x_access_token":"$http_x_access_token",'
'"referer":"$http_referer",'
'"status":"$status"}';
rpm -ivh http://172.19.30.116/mirror/elk/filebeat/7.4.0/filebeat-7.4.0-x86_64.rpm
chkconfig filebeat on
修改filebeat的配置/etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/nginx_access.log
tags: ["nginx-access"]
document_type: json-nginxaccess
tail_files: true
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
output.redis:
enabled: true
hosts: ["192.168.1.1:7000"] #这里的是自定义的REDIS服务器IP,redis端口默认是6379,请根据自己的情况修改
port: 7000
key: nginx
db: 0
datatype: list
现在我们反过来配置logstash
mkdir /srv/app/elk/logstash/config/conf.d
vim /srv/app/elk/logstash/config/conf.d/nginx-logs.conf
写入以下内容
input {
redis {
host => "192.168.1.1"
port => "7000"
key => "nginx"
data_type => "list"
threads => "5"
db => "0"
}
}
filter {
json {
source => "message"
remove_field => ["beat"]
}
geoip {
source => "clientip"
}
geoip {
source => "clientip"
target => "geoip"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
grok {
match => ["message","%{TIMESTAMP_ISO8601:isotime}"]
}
date {
locale => "en"
match => ["isotime","ISO8601"]
target => "@timestamp"
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
# remove_field => ["message"]
}
}
output {
if "nginx-access" in [tags] {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "logstash-nginx-logs-%{+YYYY.MM.dd}"
}
}
}
- 启动logstash,建议使用supervisord服务启动,建议参考: https://www.cnblogs.com/lizhaojun-ops/p/11962485.html
/srv/app/elk/logstash/bin/logstash -f /srv/app/elk/logstash/config/conf.d/nginx-logs.conf
后记
- Nginx代理kibana访问,方便添加http认证
主要的配置如下:
server {
listen 80;
server_name kibana;
access_log off;
error_log off;
location / {
auth_basic "Kibana";
auth_basic_user_file /srv/app/tengine/conf/conf.d/passwd;
proxy_pass http://127.0.0.1:5601;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
- 添加账户的脚本
/srv/app/tengine/conf/conf.d/adduser.sh
#!/bin/bash
read -p "请输入用户名: " USERNAME
read -p "请指定用户密码: " PASSWD
printf "$USERNAME:$(openssl passwd -crypt $PASSWD)
" >> passwd