创建模版
使用 devtools 创建模板，或者创建的索引以 logstash开头，确保location类型为geo_point。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
| PUT _template/nginx
{
"order": 0,
"version": 60001,
"index_patterns": [
"nginx-*"
],
"settings": {
"index": {
"number_of_shards": "1",
"refresh_interval": "5s"
}
},
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"geoip": {
"dynamic": true,
"properties": {
"ip": {
"type": "ip"
},
"latitude": {
"type": "half_float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "half_float"
}
}
},
"body_bytes": {
"type": "float"
},
"request_time": {
"type": "float"
},
"response_code": {
"type": "integer"
},
"@version": {
"type": "keyword"
}
}
},
"aliases": {}
}
|
nginx 输出 json 格式日志
可以根据自己的需求，添加删除注释
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
| log_format json '{"@timestamp":"$time_iso8601",'
'"@source":"$server_addr",'
'"hostname":"$hostname",'
'"http_host":"$host",'
'"client":"$remote_addr",'
'"request_method":"$request_method",'
'"url":"$uri",'
'"http_version":"$server_protocol",'
'"scheme":"$scheme"'
'"response_code":"$status",'
'"body_bytes":"$body_bytes_sent",'
'"request_time":"$request_time",'
'"referer":"$http_referer",'
大专栏 logstash收集nginx日志pan class="line"> '"ua":"$http_user_agent",'
#'"@version":"1",'
#'"args":"$args",'
#'"https":"$https",'
#'"http_cookie":"$http_cookie",'
#'"client_forward":"$http_x_forwarded_for",'
#'"host":"$server_addr",'
#'"upstream_host":"$upstream_addr"'
#'"upstream_status":"$upstream_status"'
#'"upstream_response_time":"$upstream_response_time"'
'}';
|
logstash 配置
1
2
3
4
5
6
7
8
9
10
11
12
| filter {
if ([fields][service] == "nginx-elastic-logs") {
json {
source => "message"
remove_field => ["message","agent","ecs"]
}
geoip {
source => "client"
target => "geoip"
}
}
}
|
输出
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| "geoip" => {
"region_name" => "Beijing",
"ip" => "123.58.107.118",
"continent_code" => "AS",
"country_name" => "China",
"country_code3" => "CN",
"location" => {
"lon" => 116.3883,
"lat" => 39.9289
},
"latitude" => 39.9289,
"country_code2" => "CN",
"region_code" => "BJ",
"longitude" => 116.3883,
"timezone" => "Asia/Shanghai"
},
|
如果觉得信息太多，可以通过 fileds 选项选择自己需要的信息,city_name, continent_code, country_code2, country_code3, country_name, dma_code, ip, latitude, longitude, postal_code, region_name and timezone
修改后的配置
1
2
3
4
5
6
7
8
9
10
11
| if ([fields][service] == "nginx-elastic-logs") {
json {
source => "message"
remove_field => ["message","agent","ecs"]
}
geoip {
source => "client"
target => "geoip"
fields => ["country_name","region_name","location","continent_code"]
}
}
|
输出
1
2
3
4
5
6
7
8
9
| "geoip" => {
"continent_code" => "AS",
"country_name" => "China",
"region_name" => "Beijing",
"location" => {
"lon" => 116.3883,
"lat" => 39.9289
}
},
|