一、首先部署logstash监控UDP514端口，新建一个配置文件cisco.conf
交换机是通过配置rsyslog服务器来将日志发送到日志服务器的，所以需要在logstash上配置rsyslog监听端口既514端口
[root@server-1 conf.d]# cd /etc/logstash/conf.d/ [root@server-1 conf.d]# vim cisco.conf
input{
syslog{
port => 514
}
}
output{
stdout{
codec => rubydebug
}
}
二、加载配置文件
[root@server-1 conf.d]# logstash -f /etc/logstash/conf.d/cisco.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2019-11-05 14:18:11.331 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2019-11-05 14:18:11.339 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ] 2019-11-05 14:18:12.069 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-11-05 14:18:12.376 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.4"}
[INFO ] 2019-11-05 14:18:12.621 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2019-11-05 14:18:13.955 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-11-05 14:18:14.536 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x24057f7e@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[INFO ] 2019-11-05 14:18:14.545 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
[INFO ] 2019-11-05 14:18:14.578 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:109] syslog - Starting syslog udp listener {:address=>"0.0.0.0:514"}
[INFO ] 2019-11-05 14:18:14.595 [Ruby-0-Thread-16: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:113] syslog - Starting syslog tcp listener {:address=>"0.0.0.0:514"}
三、测试UDP
首先看看514端口是否被监听
[root@server-1 conf.d]# netstat -tunlp|grep java tcp6 0 0 :::5002 :::* LISTEN 16102/java tcp6 0 0 172.28.18.69:9200 :::* LISTEN 18608/java tcp6 0 0 :::10001 :::* LISTEN 16102/java tcp6 0 0 172.28.18.69:9300 :::* LISTEN 18608/java tcp6 0 0 127.0.0.1:9600 :::* LISTEN 19444/java tcp6 0 0 172.28.18.69:9600 :::* LISTEN 16102/java udp 0 0 0.0.0.0:514 0.0.0.0:* 19444/java
然后，使用tcpdump命令在514端口从抓包，确认有数据包发送过来
[root@server-1 conf.d]# tcpdump -i em1 udp port 514 -c 100 -n -vvv tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size 262144 bytes
，在另外一台172.28.18.71服务器上，设置好rssyslog服务器地址，就可以发送日志到514端口了
[root@localhost ~]# vim /etc/rsyslog.conf
在"rule"下增加如下语句“*.* @@172.28.18.69“
#### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console *.* @@172.28.18.69
重启rsyslog服务
[root@localhost ~]# systemctl restart rsyslog
然后重启登录172.28.18.71，此时在172.28.18.69上tcpdump监听的514端口显示抓包的数据
[root@server-1 conf.d]# tcpdump -i em1 udp port 514 -c 100 -n -vvv tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size 262144 bytes 10:14:09.093962 IP (tos 0x0, ttl 64, id 40767, offset 0, flags [DF], proto UDP (17), length 132) 172.28.18.71.60481 > 172.28.18.69.syslog: [udp sum ok] SYSLOG, length: 104 Facility authpriv (10), Severity info (6) Msg: Nov 5 10:14:09 localhost sshd[6587]: Accepted password for root from 172.28.146.109 port 59567 ssh2 0x0000: 3c38 363e 4e6f 7620 2035 2031 303a 3134 0x0010: 3a30 3920 6c6f 6361 6c68 6f73 7420 7373 0x0020: 6864 5b36 3538 375d 3a20 4163 6365 7074 0x0030: 6564 2070 6173 7377 6f72 6420 666f 7220 0x0040: 726f 6f74 2066 726f 6d20 3137 322e 3238 0x0050: 2e31 3436 2e31 3039 2070 6f72 7420 3539 0x0060: 3536 3720 7373 6832 10:14:09.101472 IP (tos 0x0, ttl 64, id 40769, offset 0, flags [DF], proto UDP (17), length 104) 172.28.18.71.60481 > 172.28.18.69.syslog: [udp sum ok] SYSLOG, length: 76 Facility auth (4), Severity info (6) Msg: Nov 5 10:14:09 localhost systemd-logind: New session 4231 of user root. 0x0000: 3c33 383e 4e6f 7620 2035 2031 303a 3134 0x0010: 3a30 3920 6c6f 6361 6c68 6f73 7420 7379 0x0020: 7374 656d 642d 6c6f 6769 6e64 3a20 4e65 0x0030: 7720 7365 7373 696f 6e20 3432 3331 206f 0x0040: 6620 7573 6572 2072 6f6f 742e 10:14:09.101738 IP (tos 0x0, ttl 64, id 40770, offset 0, flags [DF], proto UDP (17), length 101) 172.28.18.71.60481 > 172.28.18.69.syslog: [udp sum ok] SYSLOG, length: 73 Facility daemon (3), Severity info (6) Msg: Nov 5 10:14:09 localhost systemd: Started Session 4231 of user root. 0x0000: 3c33 303e 4e6f 7620 2035 2031 303a 3134 0x0010: 3a30 3920 6c6f 6361 6c68 6f73 7420 7379 0x0020: 7374 656d 643a 2053 7461 7274 6564 2053 0x0030: 6573 7369 6f6e 2034 3233 3120 6f66 2075 0x0040: 7365 7220 726f 6f74 2e 10:14:09.102645 IP (tos 0x0, ttl 64, id 40771, offset 0, flags [DF], proto UDP (17), length 133) 172.28.18.71.60481 > 172.28.18.69.syslog: [udp sum ok] SYSLOG, length: 105 Facility authpriv (10), Severity info (6) Msg: Nov 5 10:14:09 localhost sshd[6587]: pam_unix(sshd:session): session opened for user root by (uid=0) 0x0000: 3c38 363e 4e6f 7620 2035 2031 303a 3134 0x0010: 3a30 3920 6c6f 6361 6c68 6f73 7420 7373 0x0020: 6864 5b36 3538 375d 3a20 7061 6d5f 756e 0x0030: 6978 2873 7368 643a 7365 7373 696f 6e29 0x0040: 3a20 7365 7373 696f 6e20 6f70 656e 6564 0x0050: 2066 6f72 2075 7365 7220 726f 6f74 2062 0x0060: 7920 2875 6964 3d30 29
但是查看logstash，却没有数据输出到控制台上，此时查看logstash日志
[root@server-1 log]# tail -f /home/logstash/log/logstash-plain.log
ck in start_input'"]} [2019-11-05T10:21:56,087][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:514"} [2019-11-05T10:21:56,088][WARN ][logstash.inputs.udp ] UDP listener died {:exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:200:in `bind'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:101:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:57:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507:in `block in start_input'"]} [2019-11-05T10:22:01,088][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:514"} [2019-11-05T10:22:01,090][WARN ][logstash.inputs.udp ] UDP listener died {:exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:200:in `bind'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:101:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:57:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507:in `block in start_input'"]}
从日志看出报:exception=>#<SocketError: bind: name or service not known>错误，百度一下，发现是由于514端口，必须要root用户才能启动，而logstash默认是logstash用户启动服务，所以修改logstash服务的用户
停止logstash服务
[root@server-1 conf.d]# systemctl stop logstash
修改服务配置
[root@server-1 conf.d]# vim /etc/systemd/system/logstash.service
[Unit] Description=logstash [Service] Type=simple User=logstash Group=logstash # Load env vars from /etc/default/ and /etc/sysconfig/ if they exist. # Prefixing the path with '-' makes it try to load, but if the file doesn't # exist, it continues onward. EnvironmentFile=-/etc/default/logstash EnvironmentFile=-/etc/sysconfig/logstash ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" Restart=always WorkingDirectory=/ Nice=19 LimitNOFILE=16384 [Install] WantedBy=multi-user.target
将User Group改为root
[Unit] Description=logstash [Service] Type=simple #User=logstash #Group=logstash User=root Group=root # Load env vars from /etc/default/ and /etc/sysconfig/ if they exist. # Prefixing the path with '-' makes it try to load, but if the file doesn't # exist, it continues onward. EnvironmentFile=-/etc/default/logstash EnvironmentFile=-/etc/sysconfig/logstash ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" Restart=always WorkingDirectory=/ Nice=19 LimitNOFILE=16384 [Install] WantedBy=multi-user.target
保存，重启logstash服务
[root@server-1 conf.d]# systemctl start logstash
关闭514端口监听进程，重新加载UDP监听配置文件
[root@server-1 conf.d]# logstash -f /etc/logstash/conf.d/cisco.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2019-11-05 14:18:11.331 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2019-11-05 14:18:11.339 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ] 2019-11-05 14:18:12.069 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-11-05 14:18:12.376 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.4"}
[INFO ] 2019-11-05 14:18:12.621 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2019-11-05 14:18:13.955 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-11-05 14:18:14.536 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x24057f7e@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[INFO ] 2019-11-05 14:18:14.545 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
[INFO ] 2019-11-05 14:18:14.578 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:109] syslog - Starting syslog udp listener {:address=>"0.0.0.0:514"}
[INFO ] 2019-11-05 14:18:14.595 [Ruby-0-Thread-16: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:113] syslog - Starting syslog tcp listener {:address=>"0.0.0.0:514"}
[INFO ] 2019-11-05 14:18:53.164 [Ruby-0-Thread-17: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:174] syslog - new connection {:client=>"172.28.18.71:40654"}
[INFO ] 2019-11-05 14:18:53.183 [Ruby-0-Thread-18: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:174] syslog - new connection {:client=>"172.28.18.71:40656"}
再次测试发送数据
{ "@version" => "1", "logsource" => "localhost", "priority" => 30, "facility" => 3, "host" => "172.28.18.71", "@timestamp" => 2019-11-05T06:18:53.000Z, "timestamp" => "Nov 5 14:18:53", "program" => "systemd", "facility_label" => "system", "severity" => 6, "message" => "Stopping System Logging Service... ", "severity_label" => "Informational" } { "@version" => "1", "logsource" => "localhost", "priority" => 46, "facility" => 5, "host" => "172.28.18.71", "@timestamp" => 2019-11-05T06:18:53.000Z, "timestamp" => "Nov 5 14:18:53", "program" => "rsyslogd", "facility_label" => "syslogd", "severity" => 6, "message" => "action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0-34.el7 try http://www.rsyslog.com/e/2359 ] ", "severity_label" => "Informational" }
logstash显示日志数据了，修改配置文件将日志输出到elastcisearch
output{ input{ syslog{ port => 514 } } #输出到elastcisearch output{ elasticsearch{ hosts => ["172.28.18.69:9200"] #elasticsearch服务地址 index => "system-cisco-log-%{+YYYY.MM}" #创建的索引 } }
重启加载配置文件，在elastcisearch服务器上查看索引
[root@server-1 conf.d]# curl http://172.28.18.69:9200/_cat/indices yellow open nginx-172.28.18.75-2019.11.05 WK6Zr5guQ7KSoCLPd8JjqQ 5 1 12086 0 4.5mb 4.5mb yellow open system-cisco-log-2019.11 IR__HXPvTfe3HNtQ1HOwFw 5 1 16 0 101.7kb 101.7kb green open .kibana QkF9i3nXSAKlNLMLNROM1A 1 0 4 1 23.5kb 23.5kb
已经生成了system-cisco-log-2019.11文件
四、配置交换机
这样，logstash就可以接收到交换机日志了