【原】k8s ingress-nginx 针对指定 User-Agent 爬虫进行限速

针对指定 User-Agent 进行限速

修改模板

将 ingress-nginx 模板nginx.tmpl提取出来，设置成 configmap 挂载到 ingress-nginx 中去使用

添加 map

添加如下内容到模板中

 533     map $http_user_agent $agent {
 534         default "";
 535         #~curl $http_user_agent;
 536         ~*apachebench $http_user_agent;
 537         ~*spider $http_user_agent;
 538         ~*bot  $http_user_agent;
 539         ~*slurp $http_user_agent;
 540         ~*hello $http_user_agent;
 541         ~*chrome $http_user_agent;
 542     }
 543
 544     limit_conn_zone $agent zone=conn_lyj_com:10m;
 545     limit_req_zone $agent zone=req_lyj_com:10m rate=1r/s;
 546
 547     limit_req zone=req_lyj_com burst=1 nodelay;

第一种：返回错误页面

添加错误页面

针对指定 UA 添加错误页面

添加 598-603 行内容到模板中

 594     ## start server {{ $server.Hostname }}
 595     server {
 596         server_name {{ buildServerName $server.Hostname }} {{range $server.Aliases }}{{ . }} {{ end }};
 597
 598         recursive_error_pages on;
 599         proxy_intercept_errors on;
 600         if ($http_user_agent !~ "(hello|chrome)"){
 601              set $err_page "https://www.qq.com/404/";
 602         }
 603         error_page  503 $err_page;
 604
 605         {{ if gt (len $cfg.BlockUserAgents) 0 }}
 606         if ($block_ua) {
 607            return 403;
 608         }
 609         {{ end }}
 610         {{ if gt (len $cfg.BlockReferers) 0 }}
 611         if ($block_ref) {
 612            return 403;
 613         }
 614         {{ end }}
 615
 616         {{ template "SERVER" serverConfig $all $server }}
 617
 618         {{ if not (empty $cfg.ServerSnippet) }}
 619         # Custom code snippet configured in the configuration configmap
 620         {{ $cfg.ServerSnippet }}
 621         {{ end }}

实际文件

容器中的文件内容如下

   481		## start server lyj.5179.top
   482		server {
   483			server_name lyj.5179.top ;
   484
   485			recursive_error_pages on;
   486			proxy_intercept_errors on;
   487			if ($http_user_agent !~ "(hello|chrome)"){
   488				set $err_page "https://www.qq.com/404/";
   489			}
   490			error_page  503 $err_page;
   491
   492			listen 80  ;
   493			listen 443  ssl http2 ;

测试

指定 UA 为 hello

➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080   -I -s -w '%{http_code}' -A"hello" -o /dev/null ;echo;done
200
200
302
302
302
302
302
302
302
302
302
302
302

报错日志

10.32.0.1 - - [11/Dec/2020:09:24:15 +0000] "GET / HTTP/1.1" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36" 486 0.000 [default-nginx-80] [] - - - - 621293a69b12f4b2f552ddefcfe677d5
2020/12/11 09:24:15 [error] 123#123: *1597 limiting requests, excess: 1.332 by zone "req_lyj_com", client: 10.32.0.1, server: lyj.5179.top, request: "GET / HTTP/1.1", host: "lyj.5179.top:30080"

不指定 UA

➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080   -I -s -w '%{http_code}'  -o /dev/null ;echo;done
200
200
200
200
200
200
200
200
200

第二种：不返回错误页面，返回 444

添加错误码

添加 598 行

 594     ## start server {{ $server.Hostname }}
 595     server {
 596         server_name {{ buildServerName $server.Hostname }} {{range $server.Aliases }}{{ . }} {{ end }};
 597
 598         limit_req_status 444;
 599
 600         {{ if gt (len $cfg.BlockUserAgents) 0 }}
 601         if ($block_ua) {
 602            return 403;
 603         }
 604         {{ end }}
 605         {{ if gt (len $cfg.BlockReferers) 0 }}
 606         if ($block_ref) {
 607            return 403;
 608         }

测试

指定 ua

➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080   -I -s -w '%{http_code}' -A"hello" -o /dev/null ;echo;done
200
200
000
000
000
000
000
000
...
➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080   -I  -A"hello"  ;echo;done
HTTP/1.1 200 OK
Date: Fri, 11 Dec 2020 11:59:17 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Last-Modified: Tue, 24 Nov 2020 13:02:03 GMT
ETag: "5fbd044b-264"
Accept-Ranges: bytes


HTTP/1.1 200 OK
Date: Fri, 11 Dec 2020 11:59:17 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Last-Modified: Tue, 24 Nov 2020 13:02:03 GMT
ETag: "5fbd044b-264"
Accept-Ranges: bytes


curl: (56) Recv failure: Connection reset by peer

curl: (56) Recv failure: Connection reset by peer

如果指定其他的状态码就不是 000 了

报连接被重置

日志：

10.44.0.0 - - [11/Dec/2020:12:01:03 +0000] "HEAD / HTTP/1.1" 444 0 "-" "hello" 70 0.000 [default-nginx-80] [] - - - - 5e431fb99cf77d32bc39d3657f6fc3ca
2020/12/11 12:01:03 [error] 38#38: *5185 limiting requests, excess: 1.896 by zone "req_lyj_com", client: 10.44.0.0, server: lyj.5179.top, request: "HEAD / HTTP/1.1", host: "lyj.5179.top"
10.44.0.0 - - [11/Dec/2020:12:01:03 +0000] "HEAD / HTTP/1.1" 444 0 "-" "hello" 70 0.000 [default-nginx-80] [] - - - - f58bc7feaa516f39cc9a4103fdb87ca5
2020/12/11 12:01:03 [error] 38#38: *5186 limiting requests, excess: 1.884 by zone "req_lyj_com", client: 10.44.0.0, server: lyj.5179.top, request: "HEAD / HTTP/1.1", host: "lyj.5179.top"
10.44.0.0 - - [11/Dec/2020:12:01:03 +0000] "HEAD / HTTP/1.1" 444 0 "-" "hello" 70 0.000 [default-nginx-80] [] - - - - f4d43616589ab93e4ae28c9815dd1d33
2020/12/11 12:01:03 [error] 37#37: *5187 limiting requests, excess: 1.872 by zone "req_lyj_com", client: 10.44.0.0, server: lyj.5179.top, request: "HEAD / HTTP/1.1", host: "lyj.5179.top"
10.44.0.0 - - [11/Dec/2020:12:01:03 +0000] "HEAD / HTTP/1.1" 444 0 "-" "hello" 70 0.000 [default-nginx-80] [] - - - - b68eaaa44f8ebdcaa406c651b8b0f2ad

不指定 UA

➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080   -I -s -w '%{http_code}'  -o /dev/null ;echo;done
200
200
200
200
200
200
200
200