快速搭建应用服务日志收集系统(Filebeat + ElasticSearch + kibana)

快速搭建应用服务日志收集系统(Filebeat + ElasticSearch + kibana)

概要说明

需求场景，系统环境是CentOS，多个应用部署在多台服务器上，平时查看应用日志及排查问题十分不变。索性搭建一个服务器日志收集系统，由于每日日志规模仅在G级别，所有前期暂先不搭建集群。 
技术方案是 Filebeat + ElasticSearch + kibana (日志服务器上安装ElasticSearch,Kibana,其他应用服务器上安装Filebeat); 没有考虑加上Logstash,Flume，Kafka，Redis等，一是Filebeat比较轻量级，占用资源少，且可直接将日志输出到elasticsearch,仅是方便查看线上服务日志；不需要对日志字段进行解析，所以不打算引入技术太多，增加复杂性；没用Redis,原因仅是当前已有Redis集群，但主要是交易系统使用，不想增加其额外风险。
另外，当前的选用的日志服务器内存等配置不高。不适合安装过多软件，仅此而已。


也可参考我之前写的另一篇博客：
Windows下ELK-5.4.3环境搭建 http://www.cnblogs.com/huligong1234/p/7108109.html

一、安装ElasticSearch

1.安装JDK8环境

[root@app-001 src]# cd /usr/local/src/
[root@app-001 src]# rpm -qa | grep jdk
java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el6_8.x86_64
[root@app-001 src]# rpm -e java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el6_8.x86_64
[root@app-001 src]# curl -L -O http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.rpm?AuthParam=1506173332_32b98eb52c8955419974ec3efcba2209
[root@app-001 src]# rpm -ivh jdk-8u144-linux-x64.rpm
[root@app-001 src]# java -version

2.安装ElasticSearch

[root@app-001 src]# curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.1.rpm
[root@app-001 src]# rpm -ivh elasticsearch-5.6.1.rpm
[root@app-001 src]# chkconfig --add elasticsearch

安装支持中文分词插件
[root@app-001 src]# /usr/share/elasticsearch/bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v5.6.1/elasticsearch-analysis-ik-5.6.1.zip

3.配置ElasticSearch

[root@app-001 src]# vi /etc/elasticsearch/elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /data/elasticsearch/data
#
# Path to log files:
#
#path.logs: /data/elasticsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true

#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.1.106
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
http.cors.enabled: true
http.cors.allow-origin: "*"
#http.cors.allow-origin: "esmgr.domain.com"

注意：如果CentOS版本低于7.0还需要进行如下配置调整

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

如果非内网访问，network.host需改成如下配置：

network.host: 0.0.0.0

4.启动ElasticSearch

[root@app-001 src]# service elasticsearch start
[root@app-001 src]# curl "http://192.168.1.234:9200" #查看启动情况

5.防火墙放开iptables 9200端口，允许内网其他机器访问

[root@app-001 src]# vi /etc/sysconfig/iptables
增加如下内容：
-A INPUT -s 192.168.1.0/24 -p tcp -m state --state NEW -m tcp --dport 9200 -j ACCEPT

[root@app-001 src]# service iptables restart

二、安装Filebeat

1.下载

[root@app-001 src]# curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.1-x86_64.rpm

2.安装

[root@app-001 src]# rpm -ivh filebeat-5.6.1-x86_64.rpm
[root@app-001 src]# chkconfig --add filebeat

3.配置

[root@app-001 src]# vi /etc/filebeat/filebeat.yml

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - /opt/tomcat-myapp/logs/myapp.log
    - /data/production/tomcat-myapp/logs/catalina.out
    #- c:programdataelasticsearchlogs*
  fields_under_root: true
  fields:
    log_type: myapp
  tags: ["myapp","tomcat-log"]
  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  exclude_lines: ["^DBG"]

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ["^ERR", "^WARN"]

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: [".gz$"]

  # Optional additional fields. These field can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  multiline.pattern: ^[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  multiline.match: after
  encoding: utf-8
  
- input_type: log
  paths:
    - /opt/tomcat-apiserver/logs/apiserver.log
    - /data/production/tomcat-apiserver/logs/catalina.out
  fields_under_root: true
  fields:
    log_type: apiserver
  tags: ["tomcat-log"]
  encoding: utf-8
  exclude_lines: ["^DBG"]
  multiline.pattern: ^[
  multiline.match: after
  
- input_type: log
  paths:
    - /usr/local/tengine/logs/error.log
  fields_under_root: true
  fields:
    log_type: nginx-error
  tags: ["nginx-log"]
  encoding: utf-8
  
- input_type: log
  paths:
    - /var/log/*.log
  fields_under_root: true
  fields:
    log_type: system
  tags: ["system-log"]
  encoding: utf-8
#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
fields:
    log_host: ip-106
#  env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.1.106:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

4.启动

[root@app-001 src]# /usr/bin/filebeat.sh -configtest
[root@app-001 src]# service filebeat start

[root@app-001 src]# service filebeat status #检查状态

[root@app-001 src]# tail -f /var/log/filebeat/filebeat

三、安装kibana

1.下载

[root@app-001 src]# curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-5.6.1-x86_64.rpm

2.安装

[root@app-001 src]# rpm -ivh kibana-5.6.1-x86_64.rpm
[root@app-001 src]# chkconfig --add kibana

3.配置

[root@app-001 src]# vi /etc/kibana/kibana.yml

server.port: 5601
server.host: "192.168.1.106"
elasticsearch.url: "http://192.168.1.106:9200"

4.启动

[root@app-001 src]# service kibana start

浏览器访问 http://192.168.1.106:5601/

四、安装插件

1.elasticsearch-head

1.1.安装NodeJS环境

[root@app-001 src]# curl --silent --location https://rpm.nodesource.com/setup_8.x | bash -
[root@app-001 src]# yum install -y nodejs
[root@app-001 src]# node -v 
[root@app-001 src]# npm -v

1.2.下载 elasticsearch-head

[root@app-001 src]# wget https://codeload.github.com/mobz/elasticsearch-head/zip/master
[root@app-001 src]# unzip master

1.3.修改配置

1.3.1.修改elasticsearch.yml，增加跨域的配置(需要重启es才能生效)
http.cors.enabled: true
http.cors.allow-origin: "*"

1.3.2.编辑elasticsearch-head/Gruntfile.js，修改服务器监听地址，connect节点增加hostname属性，将其值设置为*

connect: {
    server: {
        options: {
            hostname:'*',
            port: 9100,
            base: '.',
            keepalive: true
        }
    }
}

1.3.3.编辑elasticsearch-head/_site/app.js,
修改默认es地址http://localhost:9200/为http://192.168.1.106:9200/。

1.4.启动

[root@app-001 src]# npm run start
浏览器访问 http://192.168.1.106:9100/

2.bigdesk

http://bigdesk.org/

五、使用Basic Auth给ElasticSearch和Kibana配置访问权限

1.创建密码

[root@app-001 src]# htpasswd -c /usr/local/tengine/db/passwd.db loguser

2.修改Nginx配置文件nginx.conf

[root@app-001 src]# vi /usr/local/tengine/conf/nginx.conf

		server {
			listen          80;
			server_name     esmgr.domain.com;
			auth_basic "basic auth esmgr";
			auth_basic_user_file /usr/local/nginx/db/passwd.db;
			location / { 
				proxy_pass        http://192.168.1.106:9200; 
			}
			location /head/ {
				proxy_pass    http://192.168.1.106:9100/; 
			}
		}

3.更新Nginx生效

[root@app-001 src]# /usr/local/tengine/sbin/nginx -t
nginx: the configuration file /usr/local/tengine/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/tengine/conf/nginx.conf test is successful
[root@app-001 src]# /usr/local/tengine/sbin/nginx -s reload

六、CURL查询方式示例

curl -XGET "http://192.168.1.106:9200/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {"message": "JD_08011137015349778"}
  }
}'

curl -XGET "http://192.168.1.106:9200/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    
    "match": {"log_type": "myapp"}
  },"size": 5
}'

curl -XGET "http://192.168.1.106:9200/_search" -H 'Content-Type: application/json' -d'
{"query":{"bool":{"must":[{"match":{"message":"JD_08011137015349778"}}],"filter":[{"range":{"@timestamp":{"from":"now-1d","to":"now"}}}]}}}
'

curl -XGET "http://192.168.1.106:9200/_search" -H 'Content-Type: application/json' -d'
{"query":{"bool":{"must":[{"match":{"message":"JD_08011137015349778"}}],"filter":[{"range":{"@timestamp":{"gte":"1506441600000","lte":"1506527999000"}}}]}}}
'

七、Java HttpClient 方式调用简单封装示例

package org.jeedevframework.common.es;

import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;

import javax.servlet.http.HttpServletRequest;

import org.apache.commons.lang3.StringUtils;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.params.BasicHttpParams;
import org.apache.http.params.HttpConnectionParams;
import org.apache.http.util.EntityUtils;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;

import org.jeedevframework.util.DateUtil;
/**
针对如下CURL请求方式的Java封装示例
curl -XGET "http://192.168.1.106:9200/_search" -H 'Content-Type: application/json' -d'
{"query":{"bool":{"must":[{"match":{"message":"JD_08011137015349778"}}],
"filter":[{"range":{"@timestamp":{"gte":"1506441600000","lte":"1506527999000"}}}]}}}
'
 * */
public class EsQueryService {
	
	public static DefaultHttpClient httpClient = null;
	public static DefaultHttpClient getHttpClientInstance() {
		if (null == httpClient) {
			BasicHttpParams httpParams = new BasicHttpParams();  
	        HttpConnectionParams.setConnectionTimeout(httpParams, 120000);  
	        HttpConnectionParams.setSoTimeout(httpParams, 120000);  
			DefaultHttpClient httpClient = new DefaultHttpClient(httpParams);
			httpClient.getParams().setParameter("http.protocol.content-charset", "UTF-8");
			
			return httpClient;
		}
		return httpClient;
	}

	public static String query(HttpServletRequest request) throws JSONException, ParseException {
		 String keywords = request.getParameter("keywords");
		 String startTime = request.getParameter("startTime");
		 String endTime = request.getParameter("endTime");
		 int pageSize = Integer.valueOf(StringUtils.defaultIfEmpty(request.getParameter("pageSize"), "10"));
		 String log_type = StringUtils.defaultIfEmpty(request.getParameter("log_type"), "");

		if(StringUtils.isEmpty(startTime)){
			startTime = DateUtil.format(new Date(),DateUtil.C_DATE_PATTON_DEFAULT)+" 00:00:00";
		}
		if(StringUtils.isEmpty(endTime)){
			endTime = DateUtil.format(new Date(),DateUtil.C_DATE_PATTON_DEFAULT)+" 23:59:59";
		}

		long startTimeDt = new SimpleDateFormat(DateUtil.C_TIME_PATTON_DEFAULT).parse(startTime).getTime();
		long endTimeDt = new SimpleDateFormat(DateUtil.C_TIME_PATTON_DEFAULT).parse(endTime).getTime();

		DefaultHttpClient httpClient = getHttpClientInstance();

		HttpPost httpPost = new HttpPost("http://192.168.1.106:9200/_search");

		JSONObject esQueryJo = new JSONObject();
		JSONObject queryJo = new JSONObject();
		JSONObject boolJo = new JSONObject();


		//esQueryJo.put("min_score", 1.2);

		JSONArray mustJoArr = new JSONArray();
		if(StringUtils.isNotEmpty(keywords)){
			JSONObject matchJo = new JSONObject();
			matchJo.put("message", keywords);
			JSONObject matchWrapJo = new JSONObject();
			matchWrapJo.put("match", matchJo);
			mustJoArr.put(matchWrapJo);
		}

		if(StringUtils.isNotEmpty(log_type)){
			JSONObject matchJo = new JSONObject();
			matchJo.put("log_type", log_type);
			JSONObject matchWrapJo = new JSONObject();
			matchWrapJo.put("match", matchJo);
			mustJoArr.put(matchWrapJo);
		}


		JSONArray filterJoArr = new JSONArray();
		JSONObject rangeJo = new JSONObject();
		JSONObject timestampJo = new JSONObject();
		timestampJo.put("gte", startTimeDt);
		timestampJo.put("lte", endTimeDt);
		//timestampJo.put("from", "now-1d");
		//timestampJo.put("to", "now");
		rangeJo.put("@timestamp", timestampJo);
		//mustJoArr.put("match", matchJo);
		JSONObject rangeWrapJo = new JSONObject();
		rangeWrapJo.put("range", rangeJo);
		filterJoArr.put(rangeWrapJo);

		boolJo.put("must",mustJoArr);
		boolJo.put("filter",filterJoArr);

		queryJo.put("bool", boolJo);
		esQueryJo.put("query", queryJo);
		esQueryJo.put("size", pageSize);
		String esQueryString = esQueryJo.toString();
		String resultContent = "";
		if(mustJoArr.length()>0){
			StringEntity reqEntity = new StringEntity(esQueryString ,ContentType.APPLICATION_JSON);
			httpPost.setEntity(reqEntity);
			try{
				 HttpResponse resp = httpClient.execute(httpPost);
				 resultContent = EntityUtils.toString(resp.getEntity(), "UTF-8");
				 return resultContent;
			}catch(Exception e){
				e.printStackTrace();
			}finally {
			}	
		}

		return "";
	}
		

}

八、相关参考资料

 ELK 性能(1) — Logstash 性能及其替代方案
http://www.cnblogs.com/richaaaard/p/6109595.html

CentOS下RPM安装ElasticSearch
http://www.netpc.com.cn/2361.html

Elasticsearch在Centos 7上的安装与配置
https://www.biaodianfu.com/centos-7-install-elasticsearch.html


ElasticSearch 5.0.0 安装部署常见错误或问题
http://www.dajiangtai.com/community/18136.do?origin=csdn-geek&dt=1214

elasticsearch5.0启动出现的错误 
http://blog.csdn.net/qq942477618/article/details/53414983

ElasticSearch 常用的查询过滤语句
http://www.cnblogs.com/ghj1976/p/5293250.html


filebeat专题
http://www.cnblogs.com/louis2008/p/filebeat.html

filebeat.yml（中文配置详解）
http://www.cnblogs.com/zlslch/p/6622079.html

28.Filebeat的高级配置-Filebeat部分
http://blog.csdn.net/a464057216/article/details/51233375


ELK+Filebeat+Nginx集中式日志解决方案（一）
http://zhengmingjing.blog.51cto.com/1587142/1907456

ELK日志服务使用-filebeat多文件发送
http://bbotte.com/logs-service/use-elk-processing-logs-multiple-log-file-send/

初探ELK-以收集 nginx 日志为例示范搭建一个 ELK 环境的基本步骤
http://nosmoking.blog.51cto.com/3263888/1855680


filebeat+kafka+ELK5.4安装与部署
http://xiangcun168.blog.51cto.com/4788340/1933509


Filebeat5+Kafka+ELK Docker搭建日志系统
http://www.jianshu.com/p/9dfac37885cb


通过HTTP RESTful API 操作elasticsearch搜索数据 
http://blog.csdn.net/stark_summer/article/details/48830493

Elasticsearch+Logstash+Kibana教程
http://www.cnblogs.com/xing901022/p/4704319.html