部署使用elk

1.配置8台虚拟机 es[1:5],kibana,logstash,web   ip 192.168.1.61-68

2.开始配置ip和主机名

3.用ansible部署elasticsearch,并能访问网页，以下是ansible部署的yml代码

---
- hosts: es
  remote_user: root
  tasks:
    - copy:
        src: local.repo
        dest: /etc/yum.repos.d/local.repo
        owner: root
        group: root
        mode: 0644
    - name: install elasticsearch
      yum:
        name: java-1.8.0-openjdk,elasticsearch
        state: installed
    - template:
        src: elasticsearch.yml
        dest: /etc/elasticsearch/elasticsearch.yml
        owner: root
        group: root
        mode: 0644
      notify: reload elasticsearch
      tags: esconf
    - service:
        name: elasticsearch
        enabled: yes
  handlers:
    - name: reload elasticsearch
      service:
        name: elasticsearch
        state: restarted

4.以下是当前要改的 hosts

192.168.1.61 es1
192.168.1.62 es2
192.168.1.63 es3
192.168.1.64 es4
192.168.1.65 es5
192.168.1.66 kibana
192.168.1.67 logstash

5.yum源有相应的软件包和依赖包即可

6.以下是当前要改的 elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please see the documentation for further information on configuration options:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: nsd1810
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: {{ansible_hostname}}
#
# Add custom attributes to the node:
#
# node.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
# path.data: /path/to/data
#
# Path to log files:
#
# path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
# bootstrap.mlockall: true
#
# Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory
# available on the system and that the owner of the process is allowed to use this limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
# http.port: 9200
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["es1", "es2", "es3"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
# discovery.zen.minimum_master_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
# gateway.recover_after_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
#
# ---------------------------------- Various -----------------------------------
#
# Disable starting multiple nodes on a single system:
#
# node.max_local_storage_nodes: 1
#
# Require explicit names when deleting indices:
#
# action.destructive_requires_name: true

7.elasticsearch搭建完成，可以用http://192.168.1.61：9200 访问检验

8.部署插件

插件装在哪一台机器上，只能在哪台机器上使用（这里安装在es5机器上面）

1）使用远程 uri 路径可以直接安装

es5 ~]# cd /usr/share/elasticsearch/bin

es5 bin]# ./plugin install

ftp://192.168.1.254/elk/elasticsearch-head-master.zip        //安装head插件

es5 bin]# ./plugin install

ftp://192.168.1.254/elk/elasticsearch-kopf-master.zip        //安装kopf插件

es5 bin]# [root@es5 bin]# ./plugin install

ftp://192.168.1.254/elk/bigdesk-master.zip

//安装bigdesk插件    

es5 bin]# ./plugin list        //查看安装的插件

Installed plugins in /usr/share/elasticsearch/plugins:

1. - head
2. - kopf
3. - bigdesk

2）访问head插件

1. [root@room9pc01 ~]# firefox http://192.168.1.65:9200/_plugin/head

2. ）访问kopf插件 

   1. [root@room9pc01 ~]# http://192.168.1.65:9200/_plugin/kopf

4）访问bigdesk插件

1. [root@room9pc01 ~]# http://192.168.1.65:9200/_plugin/bigdesk

9：安装kibana

1）在另一台主机，配置ip为192.168.1.66，配置yum源，更改主机名

2）安装kibana

1. [root@kibana ~]# yum -y install kibana
2. [root@kibana ~]# rpm -qc kibana
3. /opt/kibana/config/kibana.yml
4. [root@kibana ~]# vim /opt/kibana/config/kibana.yml
5. 2 server.port: 5601        
6. //若把端口改为80，可以成功启动kibana，但ss时没有端口，没有监听80端口，服务里面写死了，不能用80端口，只能是5601这个端口
7. 5 server.host: "0.0.0.0"        //服务器监听地址
8. 15 elasticsearch.url: http://192.168.1.61:9200    
9. //声明地址，从哪里查，集群里面随便选一个
10. 23 kibana.index: ".kibana"    //kibana自己创建的索引
11. 26 kibana.defaultAppId: "discover"    //打开kibana页面时，默认打开的页面discover
12. 53 elasticsearch.pingTimeout: 1500    //ping检测超时时间
13. 57 elasticsearch.requestTimeout: 30000    //请求超时
14. 64 elasticsearch.startupTimeout: 5000    //启动超时
15. [root@kibana ~]# systemctl restart kibana
16. [root@kibana ~]# systemctl enable kibana
17. Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /usr/lib/systemd/system/kibana.service.
18. [root@kibana ~]# ss -antup | grep 5601 //查看监听端口

3）浏览器访问kibana，

1. [root@kibana ~]# firefox 192.168.1.66:5601

4）点击Status，查看是否安装成功，全部是绿色的对钩,说明安装成功

5）用head插件访问会有.kibana的索引信息，如图所示：

[root@es5 ~]# firefox http://192.168.1.65:9200/_plugin/head/   #插件安装再es5上

 

 10.安装jdk 和 logstash

1. [root@logstash ~]# yum -y install java-1.8.0-openjdk
2. [root@logstash ~]# yum -y install logstash
3. [root@logstash ~]# java -version

11.自己配置logstash.conf 文件，参考代码如下，input,output,filter 可以参考elastic官网https://www.elastic.co/guide/en/logstash/current/index.html

      yes的为必须选项，下面有id和example（与id在一起的地方）

      注意这个代码包含很多测试输入输出还有调用宏（

1. root@logstash ~]# cd /opt/logstash/vendor/bundle/
2. jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/
3. [root@logstash ~]# vim grok-patterns //查找COMBINEDAPACHELOG
4. COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

）查找宏路径，写入数据到/tmp/a.log，以及echo > /dev/tcp/192.168.1.67/8888 ,也可以用syslog部分，在/etc/rsyslog.conf在加上

1. local0.info @192.168.1.67:514
2. //写一个@或两个@@都可以，一个@代表udp，两个@@代表tcp

logger -p local0.info -t nds "001 elk" 然后上述操作都能被logstash，收集并处理，看到分析结果。调用宏用的是grok（因为写的正则有点复杂和头疼就用宏定义了）

input{
    stdin{ codec => "json"}
    beats{
      port => 5044
 }
file {
    path => ["/tmp/a.log","/tmp/b.log"]
    sincedb_path => "/var/lib/logstash/sincedb"
    start_position => "beginning"
    type => "testlog"
   }
 tcp {
  host => "0.0.0.0"
  port => "8888"
  type => "tcplog"
  }
 udp {
     host => "0.0.0.0"
     port => "8888"
     type => "udplog"
}

syslog {
           type => "syslog"
}

}
filter{
 grok {
  match => ["message", "%{COMBINEDAPACHELOG}"]
  }}

output{
    stdout{
    codec => "rubydebug"
   }
  elasticsearch {
    hosts => ["es1", "es2", "es3"]
    index => "weblog"
    flush_size => 2000
    idle_flush_time => 10
}
}

12.

步骤二：安装Apache服务，用filebeat收集Apache服务器的日志，存入elasticsearch

1）在之前安装了Apache的主机上面安装filebeat

1. [root@web~]# yum -y install filebeat
2. [root@web~]# vim/etc/filebeat/filebeat.yml
3. paths:
4.     - /var/log/httpd/access_log //日志的路径，短横线加空格代表yml格式
5. document_type: apachelog //文档类型
6. elasticsearch:        //加上注释
7. hosts: ["localhost:9200"]                //加上注释
8. logstash:                    //去掉注释
9. hosts: ["192.168.1.67:5044"]     //去掉注释,logstash那台主机的ip
10. [root@web ~]# systemctl start filebeat

13.然后在/etc/logstash.conf中写入beats port相关配置（filebeat是给logstash的小插件，使得各服务器不用装logstash就可以自动把数据发送给logstash服务器）

14都配置好后，就可以在logstash执行/opt/logstash/bin/logstash -f /etc/logstash/logstash.conf 

  来进行解析数据了

并且可以用netstat -antup | grep 5044 查到 两个5044的端口 （用netstat -lntup | grep 5044只有一个）

15.访问web服务器后，logstash能收集到数据，而elastic的网页可以看到weblog，在.kibana的网页输入weblog可以看到访问的条形图

 weblog图类似下面，名字不一样

.kibana图如下

不能画出饼图，数据太少，访问网页要多刷新几次，数据太少，可能看不到图片。 

上述的logstash.conf文件是用来解析httpd的log日志的如果是nginx或者是其他服务就要调用他的宏了，可以if来判断，如以下代码，提供一些思路

input{
    stdin{ codec => "json"}
    beats{
      port => 5044
 }
file {
    path => ["/tmp/a.log","/tmp/b.log"]
    sincedb_path => "/var/lib/logstash/sincedb"
    start_position => "beginning"
    type => "testlog"
   }
 tcp {
  host => "0.0.0.0"
  port => "8888"
  type => "tcplog"
  }
 udp {
     host => "0.0.0.0"
     port => "8888"
     type => "udplog"
}

syslog {
           type => "syslog"
}

}
filter{
 if [type] == "httplog"{
 grok {
  match => ["message", "%{COMBINEDAPACHELOG}"]
  }}
}

output{
    stdout{
    codec => "rubydebug"
   }
if [type] == "httplog"{
  elasticsearch {
    hosts => ["es1", "es2", "es3"]
    index => "weblog"
    flush_size => 2000
    idle_flush_time => 10
}
}}

 elk 整体思路是 客户通过访问web，然后web服务器的filebeat把数据发送给logstash，logstash解析后发送各elasticsearch检索存储，kibana上可以从elastic提取数据并有用web和图形展示出来