Docker搭建ELK笔记

ES version: 7.10.0

这里我们使用elk集成镜像，docker hub 地址：https://hub.docker.com/r/sebp/elk/

前提条件

1. Docker至少得分配3GB的内存；
2. Elasticsearch至少需要单独2G的内存；
3. 防火墙开放相关端口； 9200,5404,56001
   如果配置太差了，可以指定使用的配置参数
   vm.max_map_count至少需要262144

echo "vm.max_map_count=262144" > /etc/sysctl.conf

sysctl -p

docker-compose

version: '2'
networks:
  backend:
    driver: bridge
services:
	elk:
	  container_name: elk
	  image: sebp/elk:7.10.0
	  stdin_open: true
	  tty: true
	  restart: always
	  volumes:
	    - /opt/elk-data:/var/lib/elasticsearch
	    - /etc/localtime:/etc/localtime
	  ports:
	    - 5601:5601
	    - 9200:9200
	    - 5044:5044
	  environment:
        - ES_MIN_MEM=128m
        - ES_MAX_MEM=2048m
	  networks:
	    - backend

或者:

sudo docker run -dit --name elk 
    -p 5601:5601 
    -p 9200:9200 
    -p 5044:5044 
    -e ES_MIN_MEM=128m 
    -e ES_MAX_MEM=2048m 
    -v /opt/elk-data:/var/lib/elasticsearch 
    -v /etc/localtime:/etc/localtime 
    sebp/elk:7.10.0

---

filebeat

https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html

filebeat的版本与es的版本一致

docker pull docker.elastic.co/beats/filebeat:7.10.0

Example configuration fileedit
Download this example configuration file as a starting point:

curl -L -O https://raw.githubusercontent.com/elastic/beats/7.10/deploy/docker/filebeat.docker.yml

filebeat.docker.yml 文件的内容

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
- add_cloud_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

docker-composer.yml

version: '2'
networks:
  backend:
    driver: bridge
services:
	filebeat:
	  container_name: filebeat
	  image: docker.elastic.co/beats/filebeat:7.10.0
	  stdin_open: true
	  tty: true
	  restart: always
	  volumes:
	    - $(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml
	    - /var/lib/docker/containers:/var/lib/docker/containers
	    - /var/run/docker.sock:/var/run/docker.sock
	    - /etc/localtime:/etc/localtime
	    - /www/wwwroot/PHP/thinkphp/runtime/log/:/logs/thinkphp/
	  networks:
	    - backend

filebeat.docker.yml

# 日志输入配置
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /logs/thinkphp/*/*.log
    include_lines: ['error']
    multiline:
        pattern: '^['
        negate:  true
        match:   after
output.elasticsearch:
  hosts: ["elasticsearch:9200"]

#日志输出配置(采用 logstash 收集日志，5044为logstash端口)
#output.logstash:
#  hosts: ['elasticsearch:5044']

只提取 error 日志

---

kibana 设置显示中文

修改 kibana 安装目录下的 config/kibana.yml, 在文件末尾添加:

i18n.locale: "zh-CN"

重启 kibana

---

• 使用docker安装ELK
• ELK搭建（docker环境）
• Docker 部署ELK
• Filebeat 关键字多行匹配日志采集（multiline与include_lines） filebeat 正则表达式匹配日志

---

ES 设置密码

ES 版本: 7.10.0

1. 修改 elasticsearch 安装目录下的 config/elasticsearch.yml,在文件末尾添加,修改完之后，重启elasticsearch 服务

# discovery.type: single-node

xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true

2. 执行设置用户名和密码的命令,这里会为4个左右用户分别设置密码，elastic, kibana, logstash_system,beats_system 等

D:devopselasticsearch-7.8.0in> elasticsearch-setup-passwords interactive

Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana_system]:
Reenter password for [kibana_system]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

3. 修改密码命令如下

curl -H "Content-Type:application/json" -XPOST -u elastic 'http://127.0.0.1:9200/_xpack/security/user/elastic/_password' -d '{ "password" : "123456" }'

4. 修改 Kibana 配置文件 config/kibana.yml文件

elasticsearch.username: "elastic"
elasticsearch.password: "123456"

---

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
      - D:/workspace/lamp/thinkphp/runtime/v1/log/*/*.log
#  include_lines: ['error']
  multiline:
    pattern: '^[sd{4}-d{2}(-d{2})?'
    negate:  true
    match:   after
  tags: ["thinkphp-api"]

#output.elasticsearch:
  # Array of hosts to connect to.
#  hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"
#  username: "elastic"
#  password: "wakasann"
#  indices:
#    - index: "thinkphp-api-error"
#      when.contains:
#        tags: "thinkphp-api"


output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]
  index: "thinkphp-api-error"

filebeat 运行

filebeat.exe -c filebeat.yml -e

logstash

thinkphpapi.conf

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "wakasann"
  }
}

通过powershell 运行 logstash

 .inlogstash.bat -f .config	hinkphpapi.conf

• ElasticSearch设置用户名密码访问
• 设置elasticsearch 7.x用户名和密码
• 使用Filebeat采集日志结合logstash过滤出你想要的日志
• filebeat收集json格式的日志 可以自定义索引
• 使用Filebeat采集日志结合logstash过滤出特定格式的日志至Elasticsearch

---

• logstash日志收集之json格式 & 日志参数分离