搭建高性能web服务

要求:

安装实验环境与工具：nginx、gunicorn、gevent、curl、ab（apache 压力测试工具）等等
测试用程序：自学教程 4.2 读写字典服务 definitions_readwrite.py。请按需要在此基础上改写程序，必须使用 MongoDB 数据库。
准备 4 台电脑，记录下每台电脑配置。其中一台，安装 MongoDB ，不提供其他服务。
使用测试工具，测试单进程同步、Tornado 异步（需要改写程序和异步驱动）、GEvent 异步（需要使用wsgi方式启动）方式的性能。
使用测试工具，将前一步骤性能最好的方式，与多进程、多线程同步方式（用gunicorn wsgi方式启动）对比性能。
用 4 台电脑，组成一个性能最强的 web 服务

相关代码:

definitions_readwrite.py　　

 1 import tornado.httpserver
 2 import tornado.ioloop
 3 import tornado.options
 4 import tornado.web
 5 import tornado.wsgi
 6 
 7 from pymongo import MongoClient
 8 from tornado.options import define, options
 9 define("port", default=8001, help="run on the given port", type = int)
10 
11 class Application(tornado.web.Application):
12     def __init__(self):
13         handlers = [(r"/(w+)", WordHandler)]
14         #client = MongoClient('127.0.0.1', 27017)
15         client = MongoClient('172.18.40.194', 27017)
16         self.db = client["definitions"]
17         print self.db
18         tornado.web.Application.__init__(self, handlers, debug=True)
19 
20 class WordHandler(tornado.web.RequestHandler):
21     def get(self, word):
22         coll = self.application.db.words
23         word_doc = coll.find_one({"word": word})
24         if word_doc:
25             del word_doc["_id"]
26             self.write(word_doc)
27         else:
28             self.set_status(404)
29 
30     def post(self, word):
31         definition = self.get_argument("definition")
32         coll = self.application.db.words
33         word_doc = coll.find_one({"word": word})
34         if word_doc:
35             word_doc['definition'] = definition
36             coll.save(word_doc)
37         else:
38             word_doc = {'word': word, 'definition': definition}
39             coll.insert(word_doc)
40         del word_doc["_id"]
41         self.write(word_doc)
42 
43 # if __name__ == '__main__':
44 #     tornado.options.parse_command_line()
45 #     http_server = tornado.httpserver.HTTPServer(Application())
46 #      http_server.listen(options.port)
47 #      tornado.ioloop.IOLoop.instance().start()
48 
49 wsgi_app = tornado.wsgi.WSGIAdapter(Application())

View Code

tornado_definitions_readwirte.py

 1 import tornado.httpserver
 2 import tornado.ioloop
 3 import tornado.options
 4 import tornado.web
 5 import tornado.httpclient
 6 import tornado.gen
 7 import asyncmongo
 8 import logging
 9 import tornado.wsgi
10 import gevent
11 
12 from tornado.options import define, options
13 define("port", default=8001, help="run on the given port", type = int)
14 
15 class Application(tornado.web.Application):
16     def __init__(self):
17         handlers = [(r"/(w+)", WordHandler)]
18         #self.db = asyncmongo.Client(pool_id='mydb', host='172.18.40.194', port=27017, dbname='definitions')
19         self.db = asyncmongo.Client(pool_id='mydb', host='127.0.0.1', port=27017, dbname='definitions')
20         print self.db
21         tornado.web.Application.__init__(self, handlers, debug=True)
22 
23 class WordHandler(tornado.web.RequestHandler):
24     word = ''
25 
26     @tornado.web.asynchronous
27     def get(self, word):
28         coll = self.application.db.words
29         coll.find_one({'word':word}, callback=self.query_callback1)
30 
31     def query_callback1(self, response, error):
32         word_doc = response
33         logging.info(response)
34         if word_doc:
35             del word_doc["_id"]
36             self.write(word_doc)
37         else:
38             self.set_status(404)
39         self.finish()
40 
41     @tornado.web.asynchronous
42     def post(self, word):
43         self.word = word
44         coll = self.application.db.words
45         coll.find_one({'word': word}, callback=self.query_callback2) #
46 
47     def query_callback2(self, response, error):
48         logging.info("haha")
49         coll = self.application.db.words
50         word_doc = response
51         definition = self.get_argument("definition")
52         word = self.word
53         if word_doc:
54             responce = coll.remove({"word": word}, callback=self.delete_callback)
55         else:
56             logging.info("not exit
")
57             word_doc = {'word': word, 'definition': definition}
58             coll.insert(word_doc, callback=self.insert_callback)
59 
60     def delete_callback(self, response, error):
61         coll = self.application.db.words
62         word = self.word
63         definition = self.get_argument("definition")
64         word_doc = {'word': word, 'definition': definition}
65         coll.insert(word_doc, callback=self.insert_callback);
66 
67     def insert_callback(self, response, error):
68         word = self.word
69         definition = self.get_argument("definition")
70         word_doc = {'word': word, 'definition': definition}
71         self.write(word_doc)
72         logging.info("insert:")
73         logging.info(response)
74         self.finish()
75 
76 if __name__ == "__main__":
77      tornado.options.parse_command_line()
78      http_server = tornado.httpserver.HTTPServer(Application())
79      http_server.listen(options.port)
80      tornado.ioloop.IOLoop.instance().start()
81 #gevent.monkey.patch_all()
82 #wsgi_app = tornado.wsgi.WSGIAdapter(Application())

View Code

/etc/nginx/nginx.config

 1 #user nginx;
 2 worker_processes 4;
 3 
 4 error_log /var/log/nginx/error.log;
 5 pid /var/run/nginx.pid;
 6 
 7 events {
 8     worker_connections 4096;
 9     use epoll;
10 }
11 
12 http {
13     # Enumerate all the Tornado servers here
14     upstream frontends {
15         server 172.18.43.53:8001 weight=3;
16         server 172.18.42.169:8001 weight=4;
17         server 172.18.43.132:8001 weight=3;
18     }
19 
20     include /etc/nginx/mime.types;
21     default_type application/octet-stream;
22 
23     access_log /var/log/nginx/access.log;
24 
25     keepalive_timeout 65;
26     proxy_read_timeout 200;
27     sendfile on;
28     tcp_nopush on;
29     tcp_nodelay on;
30     gzip on;
31     gzip_min_length 1000;
32     gzip_proxied any;
33     gzip_types text/plain text/html text/css text/xml
34                application/x-javascript application/xml
35                application/atom+xml text/javascript;
36 
37     # Only retry if there was a communication error, not a timeout
38     # on the Tornado server (to avoid propagating "queries of death"
39     # to all frontends)
40     proxy_next_upstream error;
41     
42     server {
43         listen 80;
44 
45         # Allow file uploads
46         client_max_body_size 50M;
47 
48         location ^~ /static/ {
49             root /var/www;
50             if ($query_string) {
51                 expires max;
52             }
53         }
54         location = /favicon.ico {
55             rewrite (.*) /static/favicon.ico;
56         }
57         location = /robots.txt {
58             rewrite (.*) /static/robots.txt;
59         }
60 
61         location / {
62             proxy_pass_header Server;
63             proxy_set_header Host $http_host;
64             proxy_redirect off;
65             proxy_set_header X-Real-IP $remote_addr;
66             proxy_set_header X-Scheme $scheme;
67             proxy_pass http://frontends;
68         }
69     }
70 }

View Code

环境配置:

$sudo pip install gunicorn
$sudo pip install greenlet
$sudo pip install eventlet
$sudo pip install gevent
$sudo apt-get install apache2-utils (用于压力测试)

实验步骤:

UML部署图:

以下的测试均在测试量为10000，并发量为100的情况下进行

1)进行单进程同步压力测试

　　a)在A机上开启Mongodb服务, 命令如下:mongod.exe --dbpath "C:Program FilesMongoDBdatadb" ;

　　b)在B机中启动nginx；

　　c)在C机， B机，D机中都运行python definition_readwrite.py作为应用服务器

　　e)在C机中进行压力测试，命令为ab ab -n 1000 -c 10 http://localhost/ (localhost和端口号需要换成代理机的IP和端口号)

　　f)可查看测试结果

2)其他的测试也类似，其中可通过查看如下的测试结果图比较

截图:

单进程同步：

tornado异步:

Gevent

多进程同步:

多线种同步:

说明 :1)要本地运行gunicorn时的命令一般为gunicorn code:application; 这样启动gunicorn,其默认作为一个监听127.0.01:8000的web server,可以通过本机访问，如果需要通过网络来访问gunicorn服务的话，需要使用-b命令来绑定不同的地址，同时设置监听端口，也就是需要使用本机在网络中的IP地址加端口号;

　　2）修改程序为tornado异步的，不仅方式要是异步的，同时也需要使用异步的mongodb;

　　3) 用gevent运行程序，会自动将同步的程序的每个进程使其都支持IO阻塞异步的。

相关阅读:
JDBC（解析properties）
测试字符串
 TestNG基本使用
 细化到知识点的python练习题、笔试题（都说python简单，快来测试一下吧）
java操作excel（通过POI）
I/O密集型任务下，单线程、多进程、多线程、协程
 HttpClient的使用（get、post请求）
磁盘不分区，直接格式化后挂载
 添加 K8S CPU limit 会降低服务性能
 Minio设置永久下载链接
原文地址：https://www.cnblogs.com/kinthon/p/4937312.html