Logstash是一个开源的用于收集,分析和存储日志的工具。
一、环境
# dmidecode|grep "System Information" -A9|egrep "Manufacturer|Product"
Manufacturer: Dell Inc.
Product Name: PowerEdge R630
# uname -a
Linux linux-node2 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
关闭firewalld,selinux
二、安装Logstash6.2.4
2.1检查JAVA版本和环境变量
# java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
# echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64
2.2 rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
报错信息：
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
curl: (6) Could not resolve host: packages.elastic.co; Unknown error
error: https://packages.elastic.co/GPG-KEY-elasticsearch: import read failed(2).
重启系统成功，有点莫名其妙
添加repo
# vi /etc/yum.repos.d/logstash.repo
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
# yum install logstash
Loaded plugins: fastestmirror
base | 3.6 kB 00:00:00
centos-ceph-luminous | 2.9 kB 00:00:00
centos-openstack-queens | 2.9 kB 00:00:00
centos-qemu-ev | 2.9 kB 00:00:00
docker-ce-stable | 2.9 kB 00:00:00
elasticsearch-6.x | 1.3 kB 00:00:00
extras | 3.4 kB 00:00:00
logstash-6.x | 1.3 kB 00:00:00
updates | 3.4 kB 00:00:00
(1/4): logstash-6.x/primary | 67 kB 00:00:04
(2/4): centos-ceph-luminous/7/x86_64/primary_db | 115 kB 00:00:04
centos-qemu-ev/7/x86_64/primar FAILED ============================= ] 48 kB/s | 701 kB 00:00:08 ETA
http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/repodata/2dcd3ba7c05dfc6ae2e1da196d1fa38e6f417b3818f3911edf22ddabf779f273-primary.sqlite.bz2: [Errno 14] curl#7 - "Failed to connect to 2605:9000:401:102::2: Network is unreachable"
Trying other mirror.
(3/4): centos-openstack-queens/x86_64/primary_db | 902 kB 00:00:25
centos-qemu-ev/7/x86_64/primar FAILED
http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/repodata/2dcd3ba7c05dfc6ae2e1da196d1fa38e6f417b3818f3911edf22ddabf779f273-primary.sqlite.bz2: [Errno 14] curl#7 - "Failed to connect to 2605:9000:401:102::2: Network is unreachable"
Trying other mirror.
centos-qemu-ev/7/x86_64/primar FAILED
http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/repodata/2dcd3ba7c05dfc6ae2e1da196d1fa38e6f417b3818f3911edf22ddabf779f273-primary.sqlite.bz2: [Errno 14] curl#7 - "Failed to connect to 2605:9000:401:102::2: Network is unreachable"
Trying other mirror.
centos-qemu-ev/7/x86_64/primar FAILED
http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/repodata/2dcd3ba7c05dfc6ae2e1da196d1fa38e6f417b3818f3911edf22ddabf779f273-primary.sqlite.bz2: [Errno 14] curl#7 - "Failed to connect to 2605:9000:401:102::2: Network is unreachable"
Trying other mirror.
centos-qemu-ev/7/x86_64/primar FAILED
http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/repodata/2dcd3ba7c05dfc6ae2e1da196d1fa38e6f417b3818f3911edf22ddabf779f273-primary.sqlite.bz2: [Errno 14] curl#7 - "Failed to connect to 2605:9000:401:102::2: Network is unreachable"
Trying other mirror.
(4/4): centos-qemu-ev/7/x86_64/primary_db | 44 kB 00:00:03
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
logstash-6.x 180/180
Resolving Dependencies
--> Running transaction check
---> Package logstash.noarch 1:6.2.4-1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
====================================================================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================================================
Installing:
logstash noarch 1:6.2.4-1 elasticsearch-6.x 141 M
Transaction Summary
====================================================================================================================================================================
Install 1 Package
Total download size: 141 M
Installed size: 237 M
Is this ok [y/d/N]: y
Downloading packages:
logstash-6.2.4.rpm | 141 MB 00:10:34
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : 1:logstash-6.2.4-1.noarch 1/1
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Successfully created system startup script for Logstash
Verifying : 1:logstash-6.2.4-1.noarch 1/1
Installed:
logstash.noarch 1:6.2.4-1
Complete!]
# systemctl start logstash.service
# systemctl enable logstash.service
#ln -s /usr/share/logstash/bin/logstash /bin/logstash
2.3 测试logstash,一个日志存储管道有两个必需的元素，输入和输出，以及一个可选的元素，过滤器。输入插件使用来自源的数据，过滤器插件根据您的指定修改数据，并且输出插件将数据写到目的地。
#/usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-05-25 11:30:32.586 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2018-05-25 11:30:32.617 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[INFO ] 2018-05-25 11:30:32.746 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2018-05-25 11:30:32.782 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2018-05-25 11:30:33.954 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-05-25 11:30:34.057 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"935e91a7-5807-460e-9b52-ea8687f18356", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2018-05-25 11:30:34.530 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.4"}
[INFO ] 2018-05-25 11:30:34.862 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2018-05-25 11:30:36.620 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
The stdin plugin is now waiting for input:
[INFO ] 2018-05-25 11:30:36.788 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x3cadc2d2@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[INFO ] 2018-05-25 11:30:36.828 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
helloworld
{
"message" => "helloworld",
"host" => "linux-node1",
"@version" => "1",
"@timestamp" => 2018-05-25T03:31:08.872Z
}
CTRL+D终止PIPE
[INFO ] 2018-05-25 11:32:45.550 [[main]-pipeline-manager] pipeline - Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x3cadc2d2@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
2.4通过Filebeat采集APACHE WEB LOG输入配置logstash解析，logstash默认已经安装FILEBEAT。
如果你要发送跨越多行的事件需要在configuration options available in Filebeat中配置而不是使用Multiline codec plugin，使用codec会导致启动logstash报错。（比如：多行的Java stack traces消息）
#/usr/share/logstash/bin/logstash-plugin list可查看系统中已默认安装的插件，如果没有默认安装可以进行手动安装。
手动安装beat插件：
/usr/share/logstash/bin/logstash-plugin install logstash-input-beats
安装FILEBEAT
# yum install filebeat -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package filebeat.x86_64 0:6.2.4-1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
====================================================================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================================================
Installing:
filebeat x86_64 6.2.4-1 elasticsearch-6.x 12 M
Transaction Summary
====================================================================================================================================================================
Install 1 Package
Total download size: 12 M
Installed size: 49 M
Downloading packages:
filebeat-6.2.4-x86_64.rpm | 12 MB 00:00:53
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : filebeat-6.2.4-1.x86_64 1/1
Verifying : filebeat-6.2.4-1.x86_64 1/1
Installed:
filebeat.x86_64 0:6.2.4-1
Complete!
2.5配置FILEBEAT yml：
#vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- type: log
paths:
- /opt/logstash-tutorial.log #这个是下载的APACHE示例日志
output.logstash:
hosts: ["192.168.56.11:5044"]
/usr/share/filebeat/bin/ -e -c /etc/filebeat/filebeat.yml -d "publish" （一直开启）
Filebeat将尝试连接端口5044。直到logstack开始使用一个活跃的Beats插件，在这个端口上不会有任何的答案，所以你看到的关于在那个端口上连接失败的任何消息现在都是正常的。
接下来，创建一个logstash config pipeline，它使用Beats input插件从Beats接收事件。
cat /etc/logstash/first-pipeline.conf
# The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
input {
beats {
port => "5044"
}
}
output {
stdout { codec => rubydebug }
}
#/bin/logstash -f /etc/logstash/first-pipeline.conf --config.test_and_exit
--config.test_and_exit选项将解析配置文件语法并报告错误
#/bin/logstash -f /etc/logstash/first-pipeline.conf --config.reload.automatic
--config.reload.automatic 支持自动配置重载，这样您就不必每次修改配置文件时停止并重新启动日志存储
当Logstash启动后会看骊很多关于pipelines.yml的警告信息，它是用于一个logstash实例中创建安多个input pipelines
运行LOG：
/bin/logstash -f /etc/logstash/first-pipeline.conf --config.reload.automatic
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-05-25 17:08:15.239 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2018-05-25 17:08:15.256 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ] 2018-05-25 17:08:16.532 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-05-25 17:08:17.162 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.4"}
[INFO ] 2018-05-25 17:08:17.593 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2018-05-25 17:08:19.537 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2018-05-25 17:08:21.015 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2018-05-25 17:08:21.267 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2018-05-25 17:08:21.308 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x285b71d@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[INFO ] 2018-05-25 17:08:21.599 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
{
"source" => "/opt/logstash-tutorial.log",
"beat" => {
"version" => "6.2.4",
"name" => "linux-node1",
"hostname" => "linux-node1"
},
"message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"",
"host" => "linux-node1",
"@timestamp" => 2018-05-25T09:12:06.731Z,
"@version" => "1",
"prospector" => {
"type" => "log"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"offset" => 325
}
2.6引入GROK进行WEB日志解析。
# /usr/share/logstash/bin/logstash-plugin install logstash-filter-grok
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Validating logstash-filter-grok
Installing logstash-filter-grok
Installation successful
在first-pipeline.conf中添加grok配置
# vi first-pipeline.conf
input { beats { port => "5044" } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}"} } } output { stdout { codec => rubydebug } }
重新测试先删除/usr/share/filebeat/bin/data/registry文件，运行结果如下：
{
"agent" => ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"",
"verb" => "GET",
"auth" => "-",
"offset" => 325,
"@timestamp" => 2018-05-25T09:53:58.555Z,
"beat" => {
"name" => "linux-node1",
"hostname" => "linux-node1",
"version" => "6.2.4"
},
"clientip" => "83.149.9.216",
"response" => "200",
"host" => "linux-node1",
"message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"",
"bytes" => "203023",
"@version" => "1",
"source" => "/opt/logstash-tutorial.log",
"ident" => "-",
"timestamp" => "04/Jan/2015:05:13:42 +0000",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"httpversion" => "1.1",
"prospector" => {
"type" => "log"
},
"request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
"referrer" => ""http://semicomplete.com/presentations/logstash-monitorama-2013/""
}
2.7引入GEOIP插件使解析后的数据更容易搜索，示例中用来搜索IP和地理位置信息
# /usr/share/logstash/bin/logstash-plugin install logstash-filter-geoip #空格只能是一个多了不会运行
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Validating logstash-filter-geoip
Installing logstash-filter-geoip
Installation successful
修改first-pipeline.conf中添加GEOIP配置
# vi first-pipeline.conf
input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
stdout { codec => rubydebug }
}
运行结果
{
"agent" => ""Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"",
"verb" => "GET",
"geoip" => {
"continent_code" => "EU",
"city_name" => "Balham",
"country_code2" => "GB",
"region_code" => "LBH",
"latitude" => 51.4434,
"country_code3" => "GB",
"region_name" => "Lambeth",
"postal_code" => "SW12",
"location" => {
"lat" => 51.4434,
"lon" => -0.1468
},
"ip" => "86.1.76.62",
"timezone" => "Europe/London",
"country_name" => "United Kingdom",
"longitude" => -0.1468
},
"auth" => "-",
"offset" => 24464,
"@timestamp" => 2018-05-25T10:13:03.049Z,
"beat" => {
"name" => "linux-node1",
"hostname" => "linux-node1",
"version" => "6.2.4"
},
"clientip" => "86.1.76.62",
"response" => "200",
"host" => "linux-node1",
"message" => "86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"",
"bytes" => "4877",
"source" => "/opt/logstash-tutorial.log",
"@version" => "1",
"ident" => "-",
"timestamp" => "04/Jan/2015:05:30:37 +0000",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"httpversion" => "1.1",
"prospector" => {
"type" => "log"
},
"request" => "/style2.css",
"referrer" => ""http://www.semicomplete.com/projects/xdotool/""
}
2.8添加ELASTICSEARCH中添加INDEX
在上面的测试中已完成了手工测试过程并获取了我们想得到的数据格式，接下来进行在elasticsearch中添加Index，现在，web日志被分解成特定的字段， Logstash pipeline可以将数据索引到一个弹性搜索集群中。
修改first-pipeline.conf配置文件，将output设置为
# vi first-pipeline.conf
output {
stdout { codec => rubydebug }
input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => [ "192.168.56.11:9200" ]
}
}
通过这种配置，Logstash使用http协议来连接到Elasticsearch。上面的例子假设在相同的实例上运行了Logstash和Elasticsearch。您可以通过使用主机配置指定类似于hosts => [ "192.168.56.11:9200"之类的东西来指定一个远程弹搜索实例。
测试运行，先CTRL+C停止运行/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -d "publish"并删除/usr/share/filebeat/bin/data/registry
# curl -XGET '192.168.56.11:9200/logstash-2018.05.25/_search?pretty&q=response=200'
{
"took" : 2510,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 40,
"max_score" : 0.09844007,
"hits" : [
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "yD_clmMB7ZRkVSxzBppH",
"_score" : 0.09844007,
"_source" : {
"agent" : ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"",
"verb" : "GET",
"geoip" : {
"continent_code" : "EU",
"city_name" : "Moscow",
"country_code2" : "RU",
"region_code" : "MOW",
"latitude" : 55.7485,
"country_code3" : "RU",
"region_name" : "Moscow",
"postal_code" : "101194",
"location" : {
"lat" : 55.7485,
"lon" : 37.6184
},
"ip" : "83.149.9.216",
"timezone" : "Europe/Moscow",
"country_name" : "Russia",
"longitude" : 37.6184
},
"auth" : "-",
"offset" : 3584,
"@timestamp" : "2018-05-25T10:33:05.147Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "83.149.9.216",
"response" : "200",
"host" : "linux-node1",
"message" : "83.149.9.216 - - [04/Jan/2015:05:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/Dreamhost_logo.svg HTTP/1.1" 200 2126 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"",
"bytes" : "2126",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:13:46 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/presentations/logstash-monitorama-2013/images/Dreamhost_logo.svg",
"referrer" : ""http://semicomplete.com/presentations/logstash-monitorama-2013/""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "yj_clmMB7ZRkVSxzBppH",
"_score" : 0.09844007,
"_source" : {
"agent" : ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"",
"verb" : "GET",
"geoip" : {
"continent_code" : "EU",
"city_name" : "Moscow",
"country_code2" : "RU",
"region_code" : "MOW",
"latitude" : 55.7485,
"country_code3" : "RU",
"region_name" : "Moscow",
"postal_code" : "101194",
"location" : {
"lat" : 55.7485,
"lon" : 37.6184
},
"ip" : "83.149.9.216",
"timezone" : "Europe/Moscow",
"country_name" : "Russia",
"longitude" : 37.6184
},
"auth" : "-",
"offset" : 4234,
"@timestamp" : "2018-05-25T10:33:05.148Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "83.149.9.216",
"response" : "200",
"host" : "linux-node1",
"message" : "83.149.9.216 - - [04/Jan/2015:05:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/apache-icon.gif HTTP/1.1" 200 8095 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"",
"bytes" : "8095",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:13:46 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/presentations/logstash-monitorama-2013/images/apache-icon.gif",
"referrer" : ""http://semicomplete.com/presentations/logstash-monitorama-2013/""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "0D_clmMB7ZRkVSxzBppH",
"_score" : 0.09844007,
"_source" : {
"agent" : ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"",
"verb" : "GET",
"geoip" : {
"continent_code" : "EU",
"city_name" : "Moscow",
"country_code2" : "RU",
"region_code" : "MOW",
"latitude" : 55.7485,
"country_code3" : "RU",
"region_name" : "Moscow",
"postal_code" : "101194",
"location" : {
"lat" : 55.7485,
"lon" : 37.6184
},
"ip" : "83.149.9.216",
"timezone" : "Europe/Moscow",
"country_name" : "Russia",
"longitude" : 37.6184
},
"auth" : "-",
"offset" : 6167,
"@timestamp" : "2018-05-25T10:33:05.162Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "83.149.9.216",
"response" : "200",
"host" : "linux-node1",
"message" : "83.149.9.216 - - [04/Jan/2015:05:13:47 +0000] "GET /presentations/logstash-monitorama-2013/css/print/paper.css HTTP/1.1" 200 4254 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"",
"bytes" : "4254",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:13:47 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/presentations/logstash-monitorama-2013/css/print/paper.css",
"referrer" : ""http://semicomplete.com/presentations/logstash-monitorama-2013/""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "0T_clmMB7ZRkVSxzBppH",
"_score" : 0.09844007,
"_source" : {
"agent" : ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"",
"verb" : "GET",
"geoip" : {
"continent_code" : "EU",
"city_name" : "Moscow",
"country_code2" : "RU",
"region_code" : "MOW",
"latitude" : 55.7485,
"country_code3" : "RU",
"region_name" : "Moscow",
"postal_code" : "101194",
"location" : {
"lat" : 55.7485,
"lon" : 37.6184
},
"ip" : "83.149.9.216",
"timezone" : "Europe/Moscow",
"country_name" : "Russia",
"longitude" : 37.6184
},
"auth" : "-",
"offset" : 6510,
"@timestamp" : "2018-05-25T10:33:05.162Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "83.149.9.216",
"response" : "200",
"host" : "linux-node1",
"message" : "83.149.9.216 - - [04/Jan/2015:05:13:47 +0000] "GET /presentations/logstash-monitorama-2013/images/1983_delorean_dmc-12-pic-38289.jpeg HTTP/1.1" 200 220562 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"",
"bytes" : "220562",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:13:47 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/presentations/logstash-monitorama-2013/images/1983_delorean_dmc-12-pic-38289.jpeg",
"referrer" : ""http://semicomplete.com/presentations/logstash-monitorama-2013/""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "3D_clmMB7ZRkVSxzBppI",
"_score" : 0.09844007,
"_source" : {
"agent" : ""Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"",
"verb" : "GET",
"geoip" : {
"continent_code" : "NA",
"city_name" : "Leander",
"country_code2" : "US",
"region_code" : "TX",
"latitude" : 30.5423,
"country_code3" : "US",
"region_name" : "Texas",
"postal_code" : "78641",
"location" : {
"lat" : 30.5423,
"lon" : -97.9176
},
"ip" : "66.249.73.135",
"timezone" : "America/Chicago",
"country_name" : "United States",
"dma_code" : 635,
"longitude" : -97.9176
},
"auth" : "-",
"offset" : 9295,
"@timestamp" : "2018-05-25T10:33:05.179Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "66.249.73.135",
"response" : "200",
"host" : "linux-node1",
"message" : "66.249.73.135 - - [04/Jan/2015:05:15:03 +0000] "GET /blog/tags/ipv6 HTTP/1.1" 200 12251 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"",
"bytes" : "12251",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:15:03 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/blog/tags/ipv6",
"referrer" : ""-""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "4z_clmMB7ZRkVSxzBppI",
"_score" : 0.09844007,
"_source" : {
"agent" : ""Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0"",
"verb" : "GET",
"geoip" : {
"continent_code" : "AS",
"city_name" : "Kudus",
"country_code2" : "ID",
"region_code" : "JT",
"latitude" : -6.8048,
"country_code3" : "ID",
"region_name" : "Central Java",
"location" : {
"lat" : -6.8048,
"lon" : 110.8405
},
"ip" : "110.136.166.128",
"timezone" : "Asia/Jakarta",
"country_name" : "Indonesia",
"longitude" : 110.8405
},
"auth" : "-",
"offset" : 10803,
"@timestamp" : "2018-05-25T10:33:05.180Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "110.136.166.128",
"response" : "200",
"host" : "linux-node1",
"message" : "110.136.166.128 - - [04/Jan/2015:05:16:22 +0000] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0"",
"bytes" : "3638",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:16:22 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/favicon.ico",
"referrer" : ""-""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "5D_clmMB7ZRkVSxzBppI",
"_score" : 0.09844007,
"_source" : {
"agent" : ""Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0"",
"verb" : "GET",
"geoip" : {
"continent_code" : "AS",
"city_name" : "Kudus",
"country_code2" : "ID",
"region_code" : "JT",
"latitude" : -6.8048,
"country_code3" : "ID",
"region_name" : "Central Java",
"location" : {
"lat" : -6.8048,
"lon" : 110.8405
},
"ip" : "110.136.166.128",
"timezone" : "Asia/Jakarta",
"country_name" : "Indonesia",
"longitude" : 110.8405
},
"auth" : "-",
"offset" : 11021,
"@timestamp" : "2018-05-25T10:33:05.180Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "110.136.166.128",
"response" : "200",
"host" : "linux-node1",
"message" : "110.136.166.128 - - [04/Jan/2015:05:16:22 +0000] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0"",
"bytes" : "6146",
"source" : "/opt/logstash-tutorial.log",
"@version" : "1",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:16:22 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/images/jordan-80.png",
"referrer" : ""http://www.semicomplete.com/projects/xdotool/""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "6T_clmMB7ZRkVSxzBppI",
"_score" : 0.09844007,
"_source" : {
"agent" : ""-"",
"verb" : "GET",
"geoip" : {
"continent_code" : "NA",
"country_code2" : "GT",
"region_code" : "HU",
"latitude" : 15.4731,
"country_code3" : "GT",
"region_name" : "Departamento de Huehuetenango",
"location" : {
"lat" : 15.4731,
"lon" : -91.3497
},
"ip" : "200.49.190.101",
"timezone" : "America/Guatemala",
"country_name" : "Guatemala",
"longitude" : -91.3497
},
"auth" : "-",
"offset" : 12114,
"@timestamp" : "2018-05-25T10:33:05.181Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "200.49.190.101",
"response" : "200",
"host" : "linux-node1",
"message" : "200.49.190.101 - - [04/Jan/2015:05:17:39 +0000] "GET /reset.css HTTP/1.1" 200 1015 "-" "-"",
"bytes" : "1015",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:17:39 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/reset.css",
"referrer" : ""-""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "Az_clmMB7ZRkVSxzBptI",
"_score" : 0.09844007,
"_source" : {
"agent" : ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11"",
"verb" : "GET",
"geoip" : {
"continent_code" : "EU",
"city_name" : "Paris",
"country_code2" : "FR",
"region_code" : "75",
"latitude" : 48.8574,
"country_code3" : "FR",
"region_name" : "Paris",
"postal_code" : "75011",
"location" : {
"lat" : 48.8574,
"lon" : 2.3795
},
"ip" : "81.220.24.207",
"timezone" : "Europe/Paris",
"country_name" : "France",
"longitude" : 2.3795
},
"auth" : "-",
"offset" : 17730,
"@timestamp" : "2018-05-25T10:33:05.194Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "81.220.24.207",
"response" : "200",
"host" : "linux-node1",
"message" : "81.220.24.207 - - [04/Jan/2015:05:24:57 +0000] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11"",
"bytes" : "1015",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:24:57 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/reset.css",
"referrer" : ""http://www.semicomplete.com/blog/geekery/ssl-latency.html""
}
},
{
"_index" : "logstash-2018.05.25",
"_type" : "doc",
"_id" : "Dz_clmMB7ZRkVSxzBptI",
"_score" : 0.09844007,
"_source" : {
"agent" : ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"",
"verb" : "GET",
"geoip" : {
"continent_code" : "NA",
"city_name" : "Denver",
"country_code2" : "US",
"region_code" : "CO",
"latitude" : 39.7313,
"country_code3" : "US",
"region_name" : "Colorado",
"postal_code" : "80218",
"location" : {
"lat" : 39.7313,
"lon" : -104.9692
},
"ip" : "71.212.224.97",
"timezone" : "America/Denver",
"country_name" : "United States",
"dma_code" : 751,
"longitude" : -104.9692
},
"auth" : "-",
"offset" : 20410,
"@timestamp" : "2018-05-25T10:33:05.202Z",
"beat" : {
"name" : "linux-node1",
"hostname" : "linux-node1",
"version" : "6.2.4"
},
"clientip" : "71.212.224.97",
"response" : "200",
"host" : "linux-node1",
"message" : "71.212.224.97 - - [04/Jan/2015:05:27:34 +0000] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"",
"bytes" : "1015",
"@version" : "1",
"source" : "/opt/logstash-tutorial.log",
"ident" : "-",
"timestamp" : "04/Jan/2015:05:27:34 +0000",
"tags" : [
"beats_input_codec_plain_applied"
],
"httpversion" : "1.1",
"prospector" : {
"type" : "log"
},
"request" : "/reset.css",
"referrer" : ""http://www.semicomplete.com/projects/xdotool/""
}
}
]
}
}
查询可用的INDEX
# curl '192.168.56.11:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-2018.05.25 Pn0ftjJmTBy4139Os72OuA 5 1 100 0 245.3kb 245.3kb