logstash5安装并实现mariadb数据写入到elasticsearch

java环境这里默认安装了 ，一般源码安装，这里就不说了

一、安装logstash

安装logstash可以用yum安装，也可以用源码安装：

yum安装：

1.导入GPG:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

2.创建repo文件，vim /etc/yum.repos.d/logstash.repo

[logstash-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

3.安装logstash

yum install logstash

二、源码安装（推荐）

1.官网下载需要的稳定版本

2.解压：

unzip logstash-5.5.2.zip

3.移到安装目录,创建软链接，我这里安装到/opt/app目录

mv logstash-5.5.2 /opt/app/
ln -s logstash-5.5.2/ logstash

4.修改配置文件

vim /opt/app/logstash/config/logstash.yml

path.data: /opt/app/logstash/data/
pipeline.workers: 32
pipeline.output.workers: 4
pipeline.batch.size: 125
path.config: /opt/app/logstash/etc  #放配置文件
config.reload.automatic: false      #这里yes的话后面会报错，后面有提到这个问题
config.reload.interval: 3
path.logs: /data/logs/logstash

vim /opt/app/logstash/config/startup.options  修改部分选项如下：

JAVACMD=/opt/app/jdk/bin/java
LS_HOME=/opt/app/logstash
LS_JAVA_OPTS="/opt/app/jdk"
LS_PIDFILE=/opt/app/logstash/run/logstash.pid
LS_USER=logstash
LS_GROUP=logstash
LS_GC_LOG_FILE=/data/logs/logstash/gc.log

5.创建目录和用户

useradd -g 549 -u 549 logstash -s /sbin/nologin
mkdir /opt/app/logstash/run -p
mkdir /data/logs/logstash -p
mkdir /opt/app/logstash/data/ -p
mkdir /opt/app/logstash/etc -p
chown logstash:logstash /opt/app/logstash -R
chown logstash:logstash  /data/logs/logstash

6.创建启动脚本，通过systemd管理：

vim /etc/systemd/system/logstash.service

[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
EnvironmentFile=-/etc/default/logstash
#ExecStart=/opt/app/logstash/bin/logstash
ExecStart=/opt/app/logstash/bin/logstash --path.settings /opt/app/logstash/config
ExecStop=/bin/kill -s QUIT $MAINPID
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

vim /etc/default/logstash

JAVACMD="/opt/app/jdk/bin/java"
LS_HOME="/opt/app/logstash"
LS_SETTINGS_DIR="/opt/app/logstash/config"
LS_PIDFILE="/opt/app/logstash/run/logstash.pid"
LS_USER="logstash"
LS_GROUP="logstash"
LS_GC_LOG_FILE="/data/logs/logstash/gc.log"
LS_OPEN_FILES="16384"
LS_NICE="19"
SERVICE_NAME="logstash"
SERVICE_DESCRIPTION="logstash"

这样logstash就可以启动了

三、实现logstash写入mariadb数据到elasticsearch

1.安装插件：（这是一个基于jdbc的数据导入插件，可以从各种能对接此驱动的数据库导入数据到ES,还可以自行设定导入执行频率，规则等。

cd /opt/app/logstash
bin/logstash-plugin install logstash-input-jdbc

出现success表明安装成功

tips:

安装插件国外源太慢，解决办法：

（1）先安装gem

yum install gem

(2)替换国内的镜像

gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/

（3）验证是否成功

#gem sources -l
***CURRENTSOURCES***

https://gems.ruby-china.org/

（4）修改Gemfile数据源地址

cd /opt/app/logstash
修改Gemfile文件的source的值为：“https://gems.ruby-china.org/”
修改Gemfile.jruby-1.9.lock文件的remote的值为：“https://gems.ruby-china.org/

(5)执行安装插件

bin/logstash-plugin install logstash-input-jdbc

2.安装所需的java驱动包：

下载地址

解压到自定义目录，这个路径会在logstash配置文件中配置使用，我这里是：

/opt/app/logstash/lib/mysql-connector-java-5.1.44

3.生成配置文件，这里有工具可以生成

、、、、、、

tips:

logstash 启动后报错：

Logstash is not able to start since configuration auto reloading was enabled but the configuration contains plugins that don't support it. Quitting... {:pipeline_id=>"main", :plugins=>[LogStash::Inputs::Stdin]}

解决办法：

config.reload.automatic: false #关闭这个选项