nginx安装&负载均衡配置&nginx反爬虫&nginx命令

Nginx安装

wget https://nginx.org/download/nginx-1.14.0.tar.gz

tar -zxvf nginx-1.14.0.tar.gz

cd nginx-1.14.0

安装依赖库：

yum install gcc-c++

yum install -y pcre pcre-devel

yum install -y zlib zlib-devel

yum install -y openssl openssl-devel

编译安装：

./configure

make

make install

ln -s /usr/local/nginx/sbin/nginx /usr/bin/nginx

 

nginx命令：
start nginx(linux 下直接nginx启动即可）
nginx -s stop(关闭)
nginx -s reload(重启)

nginx负载均衡配置：

#user  nobody;        #全局块
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {　　　　　　　　# events块
    worker_connections  1024;
}


http {　　　　　　　　　# http块
    include       mime.types;　　# http全局块
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;
    
    upstream myserver.com{　　　　　# 负载均衡配置
        server 127.0.0.1:8080;　　　# 可以在此添加weight（权重），配置每个服务器流量的权重；例如：server 127.0.0.1:8080 weight=1;
    }

    server {　　　　# server块
        # 反爬虫    # server 全局块
        include anti_spider.conf;  # 加载反爬虫配置
        
        listen       80;
        server_name  211.67.160.21;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location ~*^.+$ {　　# location 块
            proxy_pass http://myserver.com;
            allow all;
        }
        
        error_page 404 https://www.baidu.com;

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
         error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ .php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ .php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /.ht {
        #    deny  all;
        #}
                
        location ~* ^.+.(html|jpg|jpeg|gif|png|ico|css|js)$  
        {  
            root D:/register;  
            expires 30d;  
            break;  
        }  
  
        location ~ ^/static/ {  
            root D:/register;  
            expires 30d;  
            break;  
        }   
  
        location ~ ^/ {  
            fastcgi_pass 127.0.0.1:80;  
            fastcgi_param PATH_INFO $fastcgi_script_name;  
            fastcgi_param REQUEST_METHOD $request_method;  
            fastcgi_param QUERY_STRING $query_string;  
            fastcgi_param CONTENT_TYPE $content_type;  
            fastcgi_param CONTENT_LENGTH $content_length;  
            fastcgi_param SERVER_PROTOCOL $server_protocol;  
            fastcgi_param SERVER_PORT $server_port;  
            fastcgi_param SERVER_NAME $server_name;  
            fastcgi_pass_header Authorization;  
            fastcgi_intercept_errors off;  
        }  
    }


    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}


    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;

    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}

}

• 1、全局块：配置影响nginx全局的指令。一般有运行nginx服务器的用户组，nginx进程pid存放路径，日志存放路径，配置文件引入，允许生成worker process数等。
• 2、events块：配置影响nginx服务器或与用户的网络连接。有每个进程的最大连接数，选取哪种事件驱动模型处理连接请求，是否允许同时接受多个网路连接，开启多个网络连接序列化等。
• 3、http块：可以嵌套多个server，配置代理，缓存，日志定义等绝大多数功能和第三方模块的配置。如文件引入，mime-type定义，日志自定义，是否使用sendfile传输文件，连接超时时间，单连接请求数等。
• 4、server块：配置虚拟主机的相关参数，一个http中可以有多个server。
• 5、location块：配置请求的路由，以及各种页面的处理情况。 

nginx反爬虫：

#禁止Scrapy等工具的抓取  
if ($http_user_agent ~* (Scrapy|Curl|HttpClient)) {  
     return 403;  
}  
 
#禁止指定UA及UA为空的访问  
if ($http_user_agent ~ "WinHttp|WebZIP|FetchURL|node-superagent|java/|FeedDemon|Jullo|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|Java|Feedly|Apache-HttpAsyncClient|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms|BOT/0.1|YandexBot|FlightDeckReports|Linguee Bot|^$" ) {  
     return 403;               
}  
 
#禁止非GET|HEAD|POST方式的抓取  
if ($request_method !~ ^(GET|HEAD|POST)$) {  
    return 403;  
}  
 
#屏蔽单个IP的命令是
#deny 123.45.6.7
#封整个段即从123.0.0.1到123.255.255.254的命令
#deny 123.0.0.0/8
#封IP段即从123.45.0.1到123.45.255.254的命令
#deny 124.45.0.0/16
#封IP段即从123.45.6.1到123.45.6.254的命令是
#deny 123.45.6.0/24
 
# 以下IP皆为流氓
#deny 58.95.66.0/24;

常见垃圾UA列表：

> FeedDemon 内容采集
> BOT/0.1 (BOT for JCE) sql注入
> CrawlDaddy sql注入
> Java 内容采集
> Jullo 内容采集
> Feedly 内容采集
> UniversalFeedParser 内容采集
> ApacheBench cc攻击器
> Swiftbot 无用爬虫
> YandexBot 无用爬虫
> AhrefsBot 无用爬虫
> YisouSpider 无用爬虫
> jikeSpider 无用爬虫
> MJ12bot 无用爬虫
> ZmEu phpmyadmin 漏洞扫描
> WinHttp 采集cc攻击
> EasouSpider 无用爬虫
> HttpClient tcp攻击
> Microsoft URL Control 扫描
> YYSpider 无用爬虫
> jaunty wordpress爆破扫描器
> oBot 无用爬虫
> Python-urllib 内容采集
> Indy Library 扫描
> FlightDeckReports Bot 无用爬虫
> Linguee Bot 无用爬虫