因为公司有个需求需要获取客户端的真实IP，前端是haproxy，后面是nginx，本来这个需求不难完成，但是难就难在是https请求也就是ssl
由于个人水平有限，在网上爬了很多资料，刚开始的ha是通过tcp代理443端口的，但是无法转发7层的X-Forwarded-For到后面的nginx，那么后面的ng肯定拿不到真实IP了
怎么办呢，网上爬资料
第一在HA上做ssl中断，中断后https协议就变成http协议了这样可以转发到后面nginx上，后面nginx不需要用ssl，但是问题来了，弄好页面出了点问题，并且暂时没法解决，只好暂时作罢
这条路行不通怎么办，那就用nginx做负载吧，谁让自己Low呢，nginx总在7层了吧
说干就干，安装nginx，配置下配置文件，启动，走起
主配置文件如下：
1 user www www; 2 worker_processes 20; 3 4 error_log /data/logs/nginx/error.log; 5 pid /var/run/nginx.pid; 6 7 #Specifies the value for maximum file descriptors that can be opened by this process. 8 worker_rlimit_nofile 51200; 9 10 events { 11 use epoll; 12 worker_connections 51200; 13 } 14 15 http 16 { 17 include mime.types; 18 #include proxy.conf; 19 default_type application/octet-stream; 20 21 server_names_hash_bucket_size 128; 22 client_header_buffer_size 32k; 23 large_client_header_buffers 4 32k; 24 client_max_body_size 300m; 25 #client_max_body_size 32m; 26 27 #limit_req_zone $baidu_spider zone=baidu_spider:10m rate=15r/m; 28 sendfile on; 29 tcp_nopush on; 30 31 keepalive_timeout 30; 32 33 tcp_nodelay on; 34 #ssi on; 35 #ssi_silent_errors on; 36 #ssi_types text/shtml; 37 38 fastcgi_connect_timeout 180; 39 fastcgi_send_timeout 180; 40 fastcgi_read_timeout 180; 41 fastcgi_buffer_size 128k; 42 fastcgi_buffers 8 128k; 43 fastcgi_busy_buffers_size 128k; 44 fastcgi_temp_file_write_size 128k; 45 proxy_headers_hash_max_size 51200; 46 proxy_headers_hash_bucket_size 6400; 47 gzip on; 48 gzip_min_length 2k; 49 gzip_buffers 4 16k; 50 gzip_http_version 1.0; 51 gzip_comp_level 6; 52 gzip_types text/plain application/x-javascript text/css application/xml; 53 gzip_vary on; 54 log_format access '$remote_addr,$proxy_add_x_forwarded_for,$http_x_forwarded_for,$remote_user,$time_local,$host,$request,$status,$http_referer,$HTTP_X_UP_CALLING_LINE_ID,$request_time,$http_user_agent $upstream_addr $upstream_response_time $upstream_cache_status'; 55 #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' 56 # '$status $body_bytes_sent "$http_referer" ' 57 # '"$http_x_forwarded_for" "$http_user_agent"'; 58 59 #upstream backend { 60 # server 127.0.0.1:9000 weight=5 max_fails=3 fail_timeout=30s; 61 #} 62 63 include conf.d/*; 64 65 }
然后我们配置子配置文件：
1 upstream ihouse443{ 2 server 10.0.30.37:443 max_fails=3 fail_timeout=60 weight=1; 3 server 10.0.30.38:443 max_fails=3 fail_timeout=60 weight=1; 4 server 10.0.30.39:443 max_fails=3 fail_timeout=60 weight=1; 5 6 } 7 8 9 10 server { 11 listen 443; 12 server_name www.abc.com; 13 access_log /data/logs/nginx/abc_access.log access; 14 error_log /data/logs/nginx/abc_error.log ; 15 ssl on; 16 ssl_certificate /data/ifengsite/htdocs/abc.sss.com.crt; 17 ssl_certificate_key /data/ifengsite/htdocs/abc.sss.com.key; 18 19 ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDH:AES:HIGH:!aNULL:!MD5:!ADH:!DH; 20 ssl_protocols TLSv1 TLSv1.1 TLSv1.2; 21 22 #location /stub_status { 23 # stub_status; 24 #} 25 26 27 location / { 28 proxy_pass https://ihouse443; 29 proxy_redirect off; 30 proxy_set_header Host $host; 31 proxy_set_header X-Real-IP $remote_addr; 32 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 33 #proxy_set_header X-Forwarded-For $http_x_forwarded_for; 34 proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504; 35 proxy_max_temp_file_size 0; 36 proxy_connect_timeout 90; 37 proxy_send_timeout 90; 38 proxy_read_timeout 90; 39 proxy_buffer_size 4k; 40 proxy_buffers 4 32k; 41 proxy_busy_buffers_size 64k; 42 proxy_temp_file_write_size 64k; 43 #include fastcgi_params; 44 } 45 } 46 47 server { 48 listen 443; 49 server_name *.abc.def.com; 50 access_log /data/logs/nginx/ihouse_access1.log access; 51 error_log /data/logs/nginx/ihouse_error.log ; 52 ssl on; 53 ssl_certificate /data/ifengsite/htdocs/_.sss.xxx.com.crt; 54 ssl_certificate_key /data/ifengsite/htdocs/_.sss.xxx.com.key; 55 ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDH:AES:HIGH:!aNULL:!MD5:!ADH:!DH; 56 ssl_protocols TLSv1 TLSv1.1 TLSv1.2; 57 location / { 58 proxy_pass https://ihouse443; 59 proxy_redirect off; 60 proxy_set_header Host $host; 61 proxy_set_header X-Real-IP $remote_addr; 62 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 63 #proxy_set_header X-Forwarded-For $http_x_forwarded_for; 64 proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504; 65 proxy_max_temp_file_size 0; 66 proxy_connect_timeout 90; 67 proxy_send_timeout 90; 68 proxy_read_timeout 90; 69 proxy_buffer_size 4k; 70 proxy_buffers 4 32k; 71 proxy_busy_buffers_size 64k; 72 proxy_temp_file_write_size 64k; 73 #include fastcgi_params; 74 } 75 }
然后启动nginx，去后面的ng上一看，哎怎么没真实IP呢？都是内网IP
后来在网上找了点 资料，如下解释，这是摘抄的
下面来分析请求头到达Nginx负载均衡服务器的情况；在默认情况下，Nginx并不会对X-Forwarded-For头做任何的处理，除非用户使用proxy_set_header 参数设置：
- proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
$proxy_add_x_forwarded_for变量包含客户端请求头中的"X-Forwarded-For"，与$remote_addr用逗号分开，如果没有"X-Forwarded-For" 请求头，则$proxy_add_x_forwarded_for等于$remote_addr。$remote_addr变量的值是客户端的IP。
当Nginx设置X-Forwarded-For于$proxy_add_x_forwarded_for后会有两种情况发生：
1、如果从CDN过来的请求没有设置X-Forwarded-For头（通常这种事情不会发生），而到了我们这里Nginx设置将其设置为$proxy_add_x_forwarded_for的话，X-Forwarded-For的信息应该为CDN的IP，因为相对于Nginx负载均衡来说客户端即为CDN，这样的话，后端的web程序时死活也获得不了真实用户的IP的。
2、CDN设置了X-Forwarded-For，我们这里又设置了一次，且值为$proxy_add_x_forwarded_for的话，那么X-Forwarded-For的内容变成 ”客户端IP,Nginx负载均衡服务器IP“如果是这种情况的话，那后端的程序通过X-Forwarded-For获得客户端IP，则取逗号分隔的第一项即可。
如上两点所说，如果我们知道了CDN设置了X-Forwarded-For信息，且只有客户端真实的IP的话，那么我们的Nginx负载均衡服务器可以不必理会该头，让它默认即可。
其实Nginx中还有一个$http_x_forwarded_for变量，这个变量中保存的内容就是请求中的X-Forwarded-For信息。如果后端获得X-Forwarded-For信息的程序兼容性不好的话（没有考虑到X-Forwarded-For含有多个IP的情况），最好就不要将X-Forwarded-For设置为 $proxy_add_x_forwarded_for。应该设置为$http_x_forwarded_for或者干脆不设置！
#########################################################################################################
找到了以上内容，然后就测试
首先上面配置文件蓝色内容是后来加上去的，红色内容是原本就有的
蓝色的配置加上去后，红色的注释掉后，后端的nginx就可以获取真实IP了，具体原理就是上面所说的但是我还是不太明白，后续还要好好研究下。
附上后端nginx的配置
1 user www www; 2 worker_processes 20; 3 4 error_log /data/logs/nginx/error.log; 5 pid /var/run/nginx.pid; 6 7 #Specifies the value for maximum file descriptors that can be opened by this process. 8 worker_rlimit_nofile 51200; 9 10 events { 11 use epoll; 12 worker_connections 51200; 13 } 14 15 http 16 { 17 include mime.types; 18 #include proxy.conf; 19 default_type application/octet-stream; 20 21 server_names_hash_bucket_size 128; 22 client_header_buffer_size 32k; 23 large_client_header_buffers 4 32k; 24 client_max_body_size 300m; 25 #client_max_body_size 32m; 26 27 #limit_req_zone $baidu_spider zone=baidu_spider:10m rate=15r/m; 28 sendfile on; 29 tcp_nopush on; 30 31 keepalive_timeout 30; 32 33 tcp_nodelay on; 34 #ssi on; 35 #ssi_silent_errors on; 36 #ssi_types text/shtml; 37 38 fastcgi_connect_timeout 180; 39 fastcgi_send_timeout 180; 40 fastcgi_read_timeout 180; 41 fastcgi_buffer_size 128k; 42 fastcgi_buffers 8 128k; 43 fastcgi_busy_buffers_size 128k; 44 fastcgi_temp_file_write_size 128k;47 gzip on; 48 gzip_min_length 2k; 49 gzip_buffers 4 16k; 50 gzip_http_version 1.0; 51 gzip_comp_level 6; 52 gzip_types text/plain application/x-javascript text/css application/xml; 53 gzip_vary on; 54 log_format access '$remote_addr,$http_x_forwarded_for,$remote_user,$time_local,$host,$request,$status,$http_referer,$HTTP_X_UP_CALLING_LINE_ID,$request_time,$http_user_agent $upstream_addr $upstream_response_time $upstream_cache_status'; 55 #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' 56 # '$status $body_bytes_sent "$http_referer" ' 57 # '"$http_x_forwarded_for" "$http_user_agent"'; 58 59 #upstream backend { 60 # server 127.0.0.1:9000 weight=5 max_fails=3 fail_timeout=30s; 61 #} 62 63 include conf.d/*; 64 65 }
子配置文件
upstream ihouse443{ server 10.0.30.38:34785 max_fails=3 fail_timeout=60 weight=1; server 10.0.30.38:34787 max_fails=3 fail_timeout=60 weight=1; server 10.0.10.50:34404 max_fails=3 fail_timeout=60 weight=1; } server { listen 443; server_name www.abc.com; access_log /data/logs/nginx/sss_access.log access; error_log /data/logs/nginx/ihouse_error.log ; ssl on; ssl_certificate /data/ifengsite/htdocs/ss.xx.com.crt; ssl_certificate_key /data/ifengsite/htdocs/ssxx.com.key; ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDH:AES:HIGH:!aNULL:!MD5:!ADH:!DH; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; #location /stub_status { # stub_status; #} location / { proxy_pass https://ihouse443; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; #proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-For $http_x_forwarded_for; proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504; proxy_max_temp_file_size 0; proxy_connect_timeout 90; proxy_send_timeout 90; proxy_read_timeout 90; proxy_buffer_size 4k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k; #include fastcgi_params; } }
后端nginx不需要配置
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;