【RSYSLOG】rsyslog作为日志采集器安装配置说明

RSYSLOG is the rocket-fast system for log processing.

About

由于环境基于CentOS 6.7 x64，rsyslog本身就是OS的组件，由于本文使用rsyslog作为统一日志采集器，需要与kafka进行数据交付，而原始支持kafka组件是在v8.7.0以后版本才支持，而系统自带的rsyslog是版本是v5的，因此需要对rsyslog版本进行升级。

有网的方法比较简单，通过yum命令即可，官网参考官网 http://www.rsyslog.com/rhelcentos-rpms/ 即可安装指定版本。

安装说明

由于鉴于内网环境无Internet无法使用yum进行安装，因此选择离线rpm方式升级rsyslog（rpm包下载地址：http://rpms.adiscon.com/ ）。

此处有坑：按习惯，一般使用官网提供的最新的稳定版/v8-stable/epel-7，然而由于CentOS 6.7 glibc版本为2.12而epel-7依赖的libgt-0.3.11-1.el7.x86_64、libfastjson-0.99.2-1.el7.x86_64、libestr-0.1.10-1.el7.x86_64几个组件需要 libc.so.6: version `GLIBC_2.14，CentOS 6.7升级后会提示bash版本有问题，因此此处选用/v8-stable/epel-6版本（ps：此坑是个人问题，centos6.x本来就应该使用epel-6，因此此坑可忽略）

需要下载的rpm

可在http://rpms.adiscon.com/v8-stable/epel-6/x86_64/RPMS/ 获取

• json-c-0.11-4.el6.x86_64.rpm
• libfastjson-0.99.2-1.el6.x86_64.rpm
• libgt-0.3.11-1.el6.x86_64.rpm
• libestr-0.1.10-1.el6.x86_64.rpm
• liblogging-1.0.5-1.el6.x86_64.rpm
• librdkafka1-0.8.5-0.x86_64.rpm
• rsyslog-kafka-8.19.0-1.el6.x86_64.rpm
• rsyslog-8.19.0-1.el6.x86_64.rpm

rpm安装

执行安装前 先给没个文件加上运行属性 chmod +x

rpm -ivh  json-c-0.11-4.el6.x86_64.rpm
rpm -ivh libfastjson-0.99.2-1.el6.x86_64.rpm
rpm -ivh libgt-0.3.11-1.el6.x86_64.rpm
rpm -ivh libestr-0.1.10-1.el6.x86_64.rpm
rpm -ivh liblogging-1.0.5-1.el6.x86_64.rpm
rpm -ivh librdkafka1-0.8.5-0.x86_64.rpm

#这边使用 -U 参数是为了直接升级本地rsyslog
rpm -Uvh rsyslog-8.19.0-1.el6.x86_64.rpm

#rsyslog-kafka组件安装依赖 rsyslog-8.19.0及librdkafka，该命令最后执行
rpm -ivh rsyslog-kafka-8.19.0-1.el6.x86_64.rpm

验证安装

• 安装完后执行 rsyslogd -v , 验证rsyslog是否升级成功。

# rsyslogd -v
rsyslogd 8.19.0, compiled with:
        PLATFORM:                               x86_64-redhat-linux-gnu
        PLATFORM (lsb_release -d):
        FEATURE_REGEXP:                         Yes
        GSSAPI Kerberos 5 support:              No
        FEATURE_DEBUG (debug build, slow code): No
        32bit Atomic operations supported:      Yes
        64bit Atomic operations supported:      Yes
        memory allocator:                       system default
        Runtime Instrumentation (slow code):    No
        uuid support:                           Yes
        Number of Bits in RainerScript integers: 64
 
See http://www.rsyslog.com for more information.

• 查看/lib64/rsyslog/（32位系统为 /lib/rsyslog） 中 是否存在 omkafka.so，验证rsyslog-kafka是否安装成功。

配置说明

rsyslog官网是个大坑，能路过尽量路过。

接下来说明下配置，首先声明下本文使用rsyslog的目的是作为应用的log collector，应用使用log4x(log4net:log4net.Appender.RemoteSyslogAppender, log4j:org.apache.log4j.net.SyslogAppender)，通过UDP协议将日志传输到rsyslog。

rsyslo的默认的配置文件 /etc/rsyslog.config

# rsyslog configuration file
# note that most of this config file uses old-style format,
# because it is well-known AND quite suitable for simple cases
# like we have with the default config. For more advanced
# things, RainerScript configuration is suggested.

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

module(load="imuxsock") # provides support for local system logging (e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously done by rklogd)
#module(load"immark")  # provides --MARK-- message capability

# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
#module(load="imudp") # needs to be done just once
#input(type="imudp" port="514")

# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
#module(load="imtcp") # needs to be done just once
#input(type="imtcp" port="514")


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  /var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log
 
 
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###                                      

接下来说明配置修改：

配置 input

• 开启接收UDP消息

module(load="imudp") # 加载imudp模块
input(type="imudp" port="514") # 配置UDP监听的端口号

配置 output

• 开启数据转发kafka

 module(load="omkafka") # 配置加载omkafka模块
 # 转发到kafka消息格式
template(name="rsyslogToKafka" type="list" option.json="on"){
        constant(value="{")
        #constant(value=""timestamp":"") property(name="timereported" dateFormat="rfc3339")
        constant(value=""message":"") property(name="msg" format="json")
        constant(value="","host":"") property(name="hostname")
        constant(value="","severity":"") property(name="syslogseverity")
        constant(value="","severity-text":"") property(name="syslogseverity-text")
        constant(value="","facility":"") property(name="syslogfacility-text")
        constant(value="","syslogtag":"") property(name="syslogtag")
        constant(value="","type":"application"")
        constant(value=","recieve-from":"") property(name="fromhost-ip")
        constant(value="","recieve-at":"") property(name="timegenerated" dateFormat="rfc3339")
        constant(value="","report-at":"") property(name="timereported" dateFormat="rfc3339")
        constant(value=""}")
}

# 配置kafka相关信息及模板
action( type="omkafka"
        topic="test"
        broker="11.4.74.26:9092"
        template="rsyslogToKafka"
)

• 另外一个坑:关于EscapeControlCharacters转换的问题

rsyslog接收的时候会对一些回车换行等敏感字符进行转换 比如 会被转换为#015#012，而通过log4x提交的日志如果包含Exception在kibana显示就不能正常的换行，因此关闭了rsyslog中敏感字的转换

 $EscapeControlCharactersOnReceive off