生产环境elk

生产环境elk结构如下：

 生产环境使用rsyslog来发送nginx，mysql慢日志，日常message及审计audit日志，发送到一个fluentd上，k8s集群单独使用一个fluentd来进行k8s应用的日志收集，两个fluentd将收集到的日志过滤后存储到es中，最后由kibana来进行展示。好处是rsyslog的资源占用比较少，采用日志节点自动上报的方式，系统压力与fluentd的压力会比较小，坏处就是配置比较繁琐

本篇博客先介绍日常应用及系统日志通过rsyslog发送到fluentd的方法及配置，下篇博客介绍k8s日志发送到fluentd的方法及配置

rsyslog的安装没什么介绍的，本地与网络yum均可安装：

yum install -y rsyslog

每个rsyslog的配置文件都需要做出如下修改：

# Don't log private authentication messages!　　　　注释掉第54行，然后在下面添加
*.info;mail.none;authpriv.none;cron.none;local0.none;local1.none;local2.none;local3.none;local4.none;local5.none;local6.none;local7.none           /var/log/messages

然后创建/etc/rsyslog.d/push.conf文件，每个push文件的配置都因应用日志而不同：

nginx（nginx日志需要做json化）：

$ModLoad imudp
$UDPServerRun 514

$ModLoad imfile

$InputFileName /var/log/audit/audit.log
$InputFileTag audit:
$InputFileStateFile audit.log.pos
$InputFileSeverity info
$InputFileFacility local1
$InputRunFileMonitor

$InputFileName /var/log/nginx/access.log
$InputFileTag nginx_access:
$InputFileStateFile nginx_access.log.pos
$InputFileSeverity info
$InputFileFacility local2
$InputRunFileMonitor

$InputFileName /var/log/nginx/error.log
$InputFileTag nginx_error:
$InputFileStateFile nginx_error.log.pos
$InputFileSeverity info
$InputFileFacility local3
$InputRunFileMonitor

*.info;mail.none;cron.none    @10.21.244.21:42185

mysql：

$ModLoad imudp
$UDPServerRun 514

$ModLoad imfile

$InputFileName /var/log/audit/audit.log
$InputFileTag audit:
$InputFileStateFile audit.log.pos
$InputFileSeverity info
$InputFileFacility local1
$InputRunFileMonitor

$InputFileName /data/mysql/data/slow.log
$InputFileTag mysql-slow:
$InputFileStateFile mysql-slow.log.pos
$InputFileSeverity info
$InputFileFacility local4
$InputRunFileMonitor

$InputFileName /data/mysql/data/server_audit.log
$InputFileTag mysql-audit:
$InputFileStateFile mysql-audit.log.pos
$InputFileSeverity info
$InputFileFacility local4
$InputRunFileMonitor

*.info;mail.none;cron.none    @10.21.244.21:42185

普通非业务机器：

$ModLoad imudp
$UDPServerRun 514

$ModLoad imfile

$InputFileName /var/log/audit/audit.log
$InputFileTag audit:
$InputFileStateFile audit.log.pos
$InputFileSeverity info
$InputFileFacility local1
$InputRunFileMonitor

*.info;mail.none;cron.none    @10.21.244.21:42185

修改好配置文件之后，重启rsyslog服务：

systemctl restart rsyslog

配置应用日志系统日志的fluentd的docker-compose文件：/data/fluentd/docker-compose.yml

version: "3"

services:
  fluentd:
    image: "registry.cn-hangzhou.aliyuncs.com/grammerqin-tools/fluentd"
    volumes:
      - ./config:/fluentd/etc
    ports:
      - "42185:42185/udp"
    environment:
      - FLUENTD_CONF=fluentd.conf
    container_name: fluentd

fluentd容器挂载/data/fluentd/config/fluentd.conf文件，配置如下：

<source>
  @type syslog
  port 42185
  tag rsyslog
</source>

<match rsyslog.authpriv.**>
  @type copy
  <store>
    @type elasticsearch
    hosts 10.21.244.21:9222,10.21.244.22:9222,10.21.244.23:9222
    logstash_format true
    logstash_prefix log-secure
    logstash_dateformat %Y.%m
    type_name ${tag}

    flush_interval 5s
  </store>
  <store>
    @type stdout
  </store>
</match>

<match rsyslog.local1.**>
  @type copy
  <store>
    @type elasticsearch
    hosts 10.21.244.21:9222,10.21.244.22:9222,10.21.244.23:9222
    logstash_format true
    logstash_prefix log-audit
    logstash_dateformat %Y.%m
    type_name ${tag}

    flush_interval 5s
  </store>
  <store>
    @type stdout
  </store>
</match>

<match rsyslog.local2.**>
  @type copy
  <store>
    @type elasticsearch
    hosts 10.21.244.21:9222,10.21.244.22:9222,10.21.244.23:9222
    logstash_format true
    logstash_prefix log-nginx-access
    logstash_dateformat %Y.%m
    type_name ${tag}

    flush_interval 5s
  </store>
  <store>
    @type stdout
  </store>
</match>

<match rsyslog.local3.**>
  @type copy
  <store>
    @type elasticsearch
    hosts 10.21.244.21:9222,10.21.244.22:9222,10.21.244.23:9222
    logstash_format true
    logstash_prefix log-nginx-error
    logstash_dateformat %Y.%m
    type_name ${tag}

    flush_interval 5s
  </store>
  <store>
    @type stdout
  </store>
</match>

<match rsyslog.local4.**>
  @type copy
  <store>
    @type elasticsearch
    hosts 10.21.244.21:9222,10.21.244.22:9222,10.21.244.23:9222
    logstash_format true
    logstash_prefix log-mysql
    logstash_dateformat %Y.%m
    type_name ${tag}

    flush_interval 5s
  </store>
  <store>
    @type stdout
  </store>
</match>

<match rsyslog.**>
  @type copy
  <store>
    @type elasticsearch
    hosts 10.21.244.21:9222,10.21.244.22:9222,10.21.244.23:9222
    logstash_format true
    logstash_prefix log-message
    logstash_dateformat %Y.%m
    type_name ${tag}

    flush_interval 5s
  </store>
  <store>
    @type stdout
  </store>
</match>

其中    logstash_dateformat %Y.%m表示按月进行日志索引分片，这样的话，就可以删除没有使用价值的日志索引。