CentOS 7下安装Logstash ELK Stack 日志管理系统（上）

介绍

The Elastic Stack - 它不是一个软件，而是Elasticsearch，Logstash，Kibana 开源软件的集合，对外是作为一个日志管理系统的开源方案。它可以从任何来源，任何格式进行日志搜索，分析获取数据，并实时进行展示。像盾牌（安全），监护者（警报）和Marvel（监测）一样为你的产品提供更多的可能。

Elasticsearch：搜索，提供分布式全文搜索引擎

Logstash: 日志收集，管理，存储

Kibana ：日志的过滤web 展示
Filebeat：监控日志文件、转发

测试环境规划图

环境：ip、主机名按照如上规划，系统已经 update. 所有主机时间一致。防火墙测试环境已关闭。下面是这次elk学习的部署安装

目的：通过elk 主机收集监控主要server的系统日志、以及线上应用服务日志。

安装

3.1.基础环境检查

[root@elk ~]# hostnameelk.test.com
[root@elk ~]# cat /etc/hosts127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.30.67   elk.test.com192.168.30.99   rsyslog.test.com192.168.30.64   nginx.test.com

3.2.软件包

[root@elk ~]# cd elk/[root@elk elk]# wget -c https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.3/elasticsearch-2.3.3.rpm[root@elk elk]# wget -c https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.3.2-1.noarch.rpm[root@elk elk]# wget https://download.elastic.co/kibana/kibana/kibana-4.5.1-1.x86_64.rpm[root@elk elk]# wget -c https://download.elastic.co/beats/filebeat/filebeat-1.2.3-x86_64.rpm

3.3.检查

[root@elk elk]# lselasticsearch-2.3.3.rpm  filebeat-1.2.3-x86_64.rpm  kibana-4.5.1-1.x86_64.rpm  logstash-2.3.2-1.noarch.rpm

服务器只需要安装e、l、k, 客户端只需要安装filebeat。

 3.4.安装elasticsearch，先安装jdk，elk server 需要java 开发环境支持，由于客户端上使用的是filebeat软件，它不依赖java环境，所以不需要安装。

[root@elk elk]# yum install java-1.8.0-openjdk -y[root@elk elk]# yum localinstall elasticsearch-2.3.3.rpm -y.....  Installing : elasticsearch-2.3.3-1.noarch                                                                                             1/1### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd sudo systemctl daemon-reload sudo systemctl enable elasticsearch.service### You can start elasticsearch service by executing sudo systemctl start elasticsearch.service  Verifying  : elasticsearch-2.3.3-1.noarch                                                                                             1/1Installed:  elasticsearch.noarch 0:2.3.3-1[root@elk elk]# systemctl daemon-reload[root@elk elk]# systemctl enable elasticsearchCreated symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.[root@elk elk]# systemctl start elasticsearch[root@elk elk]# systemctl status elasticsearch● elasticsearch.service - Elasticsearch   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)   Active: active (running) since Fri 2016-05-20 15:38:35 CST; 12s ago     Docs: http://www.elastic.co  Process: 10428 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS) Main PID: 10430 (java)   CGroup: /system.slice/elasticsearch.service           └─10430 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancy...May 20 15:38:38 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:38,279][INFO ][env                      ] [James Howlett] heap...[true]May 20 15:38:38 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:38,279][WARN ][env                      ] [James Howlett] max ...65536]May 20 15:38:41 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:41,726][INFO ][node                     ] [James Howlett] initializedMay 20 15:38:41 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:41,726][INFO ][node                     ] [James Howlett] starting ...May 20 15:38:41 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:41,915][INFO ][transport                ] [James Howlett] publ...:9300}May 20 15:38:41 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:41,920][INFO ][discovery                ] [James Howlett] elas...xx35hwMay 20 15:38:45 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:45,099][INFO ][cluster.service          ] [James Howlett] new_...eived)May 20 15:38:45 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:45,164][INFO ][gateway                  ] [James Howlett] reco..._stateMay 20 15:38:45 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:45,185][INFO ][http                     ] [James Howlett] publ...:9200}May 20 15:38:45 elk.test.com elasticsearch[10430]: [2016-05-20 15:38:45,185][INFO ][node                     ] [James Howlett] startedHint: Some lines were ellipsized, use -l to show in full.

 检查服务

[root@elk elk]# rpm -qc elasticsearch/etc/elasticsearch/elasticsearch.yml/etc/elasticsearch/logging.yml/etc/init.d/elasticsearch/etc/sysconfig/elasticsearch/usr/lib/sysctl.d/elasticsearch.conf/usr/lib/systemd/system/elasticsearch.service/usr/lib/tmpfiles.d/elasticsearch.conf
[root@elk elk]# netstat -nltp | grep java
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      10430/java
tcp6       0      0 ::1:9200                :::*                    LISTEN      10430/java
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      10430/java
tcp6       0      0 ::1:9300                :::*                    LISTEN      10430/java

修改防火墙，将9200、9300 端口对外开放

[root@elk elk]# firewall-cmd --permanent --add-port={9200/tcp,9300/tcp}
success
[root@elk elk]# firewall-cmd --reload
success
[root@elk elk]# firewall-cmd  --list-all
public (default, active)
 interfaces: eno16777984 eno33557248
 sources:
 services: dhcpv6-client ssh
 ports: 9200/tcp 9300/tcp
 masquerade: no
 forward-ports:
 icmp-blocks:
 rich rules:

3.5 安装kibana

[root@elk elk]# yum localinstall kibana-4.5.1-1.x86_64.rpm –y
[root@elk elk]# systemctl enable kibana
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /usr/lib/systemd/system/kibana.service.
[root@elk elk]# systemctl start kibana

[root@elk elk]# systemctl status kibana
● kibana.service - no description given
  Loaded: loaded (/usr/lib/systemd/system/kibana.service; enabled; vendor preset: disabled)
  Active: active (running) since Fri 2016-05-20 15:49:02 CST; 20s ago
Main PID: 11260 (node)
  CGroup: /system.slice/kibana.service
          └─11260 /opt/kibana/bin/../node/bin/node /opt/kibana/bin/../src/cli

May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:elasticsearch...May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:kbn_vi...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:markdo...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:metric...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:spyMod...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:status...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:table_...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["listening","info"],"pi...:5601"}
May 20 15:49:10 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:10+00:00","tags":["status","plugin:elasticsearch...May 20 15:49:14 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:14+00:00","tags":["status","plugin:elasti...found"}
Hint: Some lines were ellipsized, use -l to show in full.

检查kibana服务运行（Kibana默认 进程名：node ，端口5601）

[root@elk elk]# netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      909/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1595/master
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      11260/node

修改防火墙，对外开放tcp/5601

[root@elk elk]# firewall-cmd --permanent --add-port=5601/tcp
Success
[root@elk elk]# firewall-cmd --reload
success
[root@elk elk]# firewall-cmd --list-all
public (default, active)
 interfaces: eno16777984 eno33557248
 sources:
 services: dhcpv6-client ssh
 ports: 9200/tcp 9300/tcp 5601/tcp
 masquerade: no
 forward-ports:
 icmp-blocks:
 rich rules:

3.5 安装kibana

[root@elk elk]# yum localinstall kibana-4.5.1-1.x86_64.rpm –y
[root@elk elk]# systemctl enable kibana
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /usr/lib/systemd/system/kibana.service.
[root@elk elk]# systemctl start kibana

[root@elk elk]# systemctl status kibana
● kibana.service - no description given
  Loaded: loaded (/usr/lib/systemd/system/kibana.service; enabled; vendor preset: disabled)
  Active: active (running) since Fri 2016-05-20 15:49:02 CST; 20s ago
Main PID: 11260 (node)
  CGroup: /system.slice/kibana.service
          └─11260 /opt/kibana/bin/../node/bin/node /opt/kibana/bin/../src/cli

May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:elasticsearch...May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:kbn_vi...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:markdo...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:metric...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:spyMod...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:status...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["status","plugin:table_...lized"}
May 20 15:49:05 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:05+00:00","tags":["listening","info"],"pi...:5601"}
May 20 15:49:10 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:10+00:00","tags":["status","plugin:elasticsearch...May 20 15:49:14 elk.test.com kibana[11260]: {"type":"log","@timestamp":"2016-05-20T07:49:14+00:00","tags":["status","plugin:elasti...found"}
Hint: Some lines were ellipsized, use -l to show in full.

检查kibana服务运行（Kibana默认 进程名：node ，端口5601）

[root@elk elk]# netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      909/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1595/master
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      11260/node

修改防火墙，对外开放tcp/5601

[root@elk elk]# firewall-cmd --permanent --add-port=5601/tcp
Success
[root@elk elk]# firewall-cmd --reload
success
[root@elk elk]# firewall-cmd --list-all
public (default, active)
 interfaces: eno16777984 eno33557248
 sources:
 services: dhcpv6-client ssh
 ports: 9200/tcp 9300/tcp 5601/tcp
 masquerade: no
 forward-ports:
 icmp-blocks:
 rich rules:

阅读

投诉

精选留言

该文章作者已设置需关注才可以留言

写留言

该文章作者已设置需关注才可以留言

写留言

加载中

以上留言由公众号筛选后显示

了解留言功能详情

微信扫一扫
关注该公众号

来自为知笔记(Wiz)