一、http协议原理
http是无状态的应用层协议
二、urllib、urllib2、requests
在python2中，有urllib和urlib2两个包，python3全变成urllib了
urllib和urllib2是相互独立的模块,比如，urllib不能伪造user agent，但是请求参数的构建，需要使用到urllib,所有经常是两者配合使用的;
requests使用了urlib3(多次请求重复使用一个socket)
三、环境准备
http://httpbin.org/
如果将该站点搭建在本地，使用pip安装就可以了
pip install gunicorn httpbin
启动:
gunicorn httpbin:app --bind 0.0.0.0:8000
发送一个http的get请求，获取本机的IP地址
import urllib2
import urllib
URL_IP='http://192.168.74.20:8000/ip'
def use_simple_urllib2():
response=urllib2.urlopen(URL_IP)
print ">>>>Response Headers:"
print response.info()
print ''.join([line for line in response.readlines()])
if __name__ == '__main__':
print ">>>Use Simple urllib2"
use_simple_urllib2()
结果为:
>>>Use Simple urllib2
>>>>Response Headers:
Server: gunicorn/19.7.1
Date: Sat, 08 Apr 2017 13:06:18 GMT
Connection: close
Content-Type: application/json
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Length: 31
{
"origin": "192.168.74.1"
}
使用urllib构建请求参数,使用urllib2发送请求
def use_params_urllib2():
"""
构建请求参数
:return:
"""
params = urllib.urlencode({'param1':'hello','param2':'world'})
print 'Request Params:'
print params
#发送请求
response=urllib2.urlopen('?'.join([URL_GET,'%s'])%params)
#处理响应
print ">>>>Response Headers:"
print response.info()
print ">>>Status code"
print response.getcode()
print ''.join([line for line in response.readlines()])
if __name__ == '__main__':
print ">>>Use Simple urllib2"
use_simple_urllib2()
print ">>>Use params urllib2"
use_params_urllib2()
结果:
>>>Use params urllib2
Request Params:
param2=world¶m1=hello
>>>>Response Headers:
Server: gunicorn/19.7.1
Date: Sat, 08 Apr 2017 13:22:05 GMT
Connection: close
Content-Type: application/json
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Length: 322
>>>Status code
200
{
"args": {
"param1": "hello",
"param2": "world"
},
"headers": {
"Accept-Encoding": "identity",
"Connection": "close",
"Host": "192.168.74.20:8000",
"User-Agent": "Python-urllib/2.7"
},
"origin": "192.168.74.1",
"url": "http://192.168.74.20:8000/get?param2=world¶m1=hello"
}
使用requests模块实现上述功能
#!/usr/bin/env python
# _*_ coding:utf-8 _*_
__author__ = 'Charles Chang'
import requests
import urllib2
import urllib
URL_IP='http://192.168.74.20:8000/ip'
URL_GET='http://192.168.74.20:8000/get'
def use_simple_requests():
response=requests.get(URL_IP)
print ">>>>Response Headers:"
print response.headers
print '>>>>Response Body:'
print response.text
def use_params_resuqets():
"""
构建请求参数
:return:
"""
params = {'param1':'hello','param2':'world'}
print 'Request Params:'
print params
#发送请求
response=requests.get(URL_IP,params=params)
#处理响应
print ">>>>Response Headers:"
print response.headers
print ">>>Status code"
print response.status_code
print response.reason
print ">>>>Response Body:"
print response.json()
if __name__ == '__main__':
print ">>>Use Simple urllib2"
use_simple_requests()
print
print ">>>Use params requests"
use_params_resuqets()
结果：
>>>Use Simple urllib2
>>>>Response Headers:
{'Content-Length': '31', 'Server': 'gunicorn/19.7.1', 'Connection': 'close', 'Access-Control-Allow-Credentials': 'true', 'Date': 'Sat, 08 Apr 2017 14:29:38 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json'}
>>>>Response Body:
{
"origin": "192.168.74.1"
}
>>>Use params requests
Request Params:
{'param2': 'world', 'param1': 'hello'}
>>>>Response Headers:
{'Content-Length': '31', 'Server': 'gunicorn/19.7.1', 'Connection': 'close', 'Access-Control-Allow-Credentials': 'true', 'Date': 'Sat, 08 Apr 2017 14:29:38 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json'}
>>>Status code
200
OK
>>>>Response Body:
{u'origin': u'192.168.74.1'}
四、github api，requests 请求方法
https://developer.github.com/v3/
GET:查看资源
POST:增加资源
PUT：修改资源
DELETE：删除资源
HEAD:查看响应头
OPTIONS:查看可用请求方法
PATCH:使用少量的json格式的数据去更新数据的方式;
| Verb | Description |
|---|---|
HEAD |
Can be issued against any resource to get just the HTTP header info. |
GET |
Used for retrieving resources. |
POST |
Used for creating resources. |
PATCH |
Used for updating resources with partial JSON data. For instance, an Issue resource hastitle and body attributes. A PATCH request may accept one or more of the attributes to update the resource. PATCH is a relatively new and uncommon HTTP verb, so resource endpoints also accept POST requests. |
PUT |
Used for replacing resources or collections. For PUT requests with no body attribute, be sure to set the Content-Length header to zero. |
DELETE |
Used for deleting resources. |
requests请求方法调用github api
import json
import requests
URL='https://api.github.com'
def build_uri(endpoint):
return '/'.join([URL,endpoint])
def better_print(json_str):
return json.dumps(json.loads(json_str),indent=4)
def request_method():
response=requests.get(build_uri('users/myuser'),auth=('myuser','mypassword'))
print response.text
print better_print(response.text)
if __name__ == '__main__':
request_method()
结果:
{"login":"CharlesQQ","id":18431718,"avatar_url":"https://avatars2.githubusercontent.com/u/18431718?v=3","gravatar_id":"","url":"https://api.github.com/users/CharlesQQ","html_url":"https://github.com/CharlesQQ","followers_url":"https://api.github.com/users/CharlesQQ/followers","following_url":"https://api.github.com/users/CharlesQQ/following{/other_user}","gists_url":"https://api.github.com/users/CharlesQQ/gists{/gist_id}","starred_url":"https://api.github.com/users/CharlesQQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CharlesQQ/subscriptions","organizations_url":"https://api.github.com/users/CharlesQQ/orgs","repos_url":"https://api.github.com/users/CharlesQQ/repos","events_url":"https://api.github.com/users/CharlesQQ/events{/privacy}","received_events_url":"https://api.github.com/users/CharlesQQ/received_events","type":"User","site_admin":false,"name":null,"company":null,"blog":null,"location":null,"email":null,"hireable":null,"bio":null,"public_repos":7,"public_gists":0,"followers":0,"following":8,"created_at":"2016-04-13T00:58:55Z","updated_at":"2017-04-08T14:36:42Z","private_gists":0,"total_private_repos":0,"owned_private_repos":0,"disk_usage":14729,"collaborators":0,"two_factor_authentication":false,"plan":{"name":"free","space":976562499,"collaborators":0,"private_repos":0}}
{
"disk_usage": 14729,
"private_gists": 0,
"public_repos": 7,
"site_admin": false,
"subscriptions_url": "https://api.github.com/users/CharlesQQ/subscriptions",
"gravatar_id": "",
"hireable": null,
"id": 18431718,
"followers_url": "https://api.github.com/users/CharlesQQ/followers",
"following_url": "https://api.github.com/users/CharlesQQ/following{/other_user}",
"collaborators": 0,
"total_private_repos": 0,
"blog": null,
"followers": 0,
"location": null,
"type": "User",
"email": null,
"bio": null,
"gists_url": "https://api.github.com/users/CharlesQQ/gists{/gist_id}",
"owned_private_repos": 0,
"company": null,
"events_url": "https://api.github.com/users/CharlesQQ/events{/privacy}",
"html_url": "https://github.com/CharlesQQ",
"updated_at": "2017-04-08T14:36:42Z",
"plan": {
"collaborators": 0,
"name": "free",
"private_repos": 0,
"space": 976562499
},
"received_events_url": "https://api.github.com/users/CharlesQQ/received_events",
"starred_url": "https://api.github.com/users/CharlesQQ/starred{/owner}{/repo}",
"public_gists": 0,
"name": null,
"organizations_url": "https://api.github.com/users/CharlesQQ/orgs",
"url": "https://api.github.com/users/CharlesQQ",
"created_at": "2016-04-13T00:58:55Z",
"avatar_url": "https://avatars2.githubusercontent.com/u/18431718?v=3",
"repos_url": "https://api.github.com/users/CharlesQQ/repos",
"following": 8,
"login": "CharlesQQ",
"two_factor_authentication": false
}
五、带参数的请求
方式1:
search_product.html?cat=19283377&...
params: requests.get(url,params={'key1':''value1})
方式2：表单数据提交
Content-Type: application/x-www-form-urlencoded
requests.post(url,data={'key1':'value1','keys':'value2'})
方式3：json参数提交
Content-type:application/json
requests.post(url,json={'key1':'value1','key2':'value2'})
使用params获取user信息
def params_requests():
response=requests.get(build_uri('users'),params={'since':11})
print better_print(response.text)
print response.request.headers
print response.url
if __name__ == '__main__':
params_requests()
结果：
{
"following_url": "https://api.github.com/users/tomtt/following{/other_user}",
"events_url": "https://api.github.com/users/tomtt/events{/privacy}",
"organizations_url": "https://api.github.com/users/tomtt/orgs",
"url": "https://api.github.com/users/tomtt",
"gists_url": "https://api.github.com/users/tomtt/gists{/gist_id}",
"html_url": "https://github.com/tomtt",
"subscriptions_url": "https://api.github.com/users/tomtt/subscriptions",
"avatar_url": "https://avatars1.githubusercontent.com/u/31?v=3",
"repos_url": "https://api.github.com/users/tomtt/repos",
"received_events_url": "https://api.github.com/users/tomtt/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/tomtt/starred{/owner}{/repo}",
"site_admin": false,
"login": "tomtt",
"type": "User",
"id": 31,
"followers_url": "https://api.github.com/users/tomtt/followers"
},
{
"following_url": "https://api.github.com/users/railsjitsu/following{/other_user}",
"events_url": "https://api.github.com/users/railsjitsu/events{/privacy}",
"organizations_url": "https://api.github.com/users/railsjitsu/orgs",
"url": "https://api.github.com/users/railsjitsu",
"gists_url": "https://api.github.com/users/railsjitsu/gists{/gist_id}",
"html_url": "https://github.com/railsjitsu",
"subscriptions_url": "https://api.github.com/users/railsjitsu/subscriptions",
"avatar_url": "https://avatars1.githubusercontent.com/u/32?v=3",
"repos_url": "https://api.github.com/users/railsjitsu/repos",
"received_events_url": "https://api.github.com/users/railsjitsu/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/railsjitsu/starred{/owner}{/repo}",
"site_admin": false,
"login": "railsjitsu",
"type": "User",
"id": 32,
"followers_url": "https://api.github.com/users/railsjitsu/followers"
},
{
"following_url": "https://api.github.com/users/nitay/following{/other_user}",
"events_url": "https://api.github.com/users/nitay/events{/privacy}",
"organizations_url": "https://api.github.com/users/nitay/orgs",
"url": "https://api.github.com/users/nitay",
"gists_url": "https://api.github.com/users/nitay/gists{/gist_id}",
"html_url": "https://github.com/nitay",
"subscriptions_url": "https://api.github.com/users/nitay/subscriptions",
"avatar_url": "https://avatars1.githubusercontent.com/u/34?v=3",
"repos_url": "https://api.github.com/users/nitay/repos",
"received_events_url": "https://api.github.com/users/nitay/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/nitay/starred{/owner}{/repo}",
"site_admin": false,
"login": "nitay",
"type": "User",
"id": 34,
"followers_url": "https://api.github.com/users/nitay/followers"
},
{
"following_url": "https://api.github.com/users/kevwil/following{/other_user}",
"events_url": "https://api.github.com/users/kevwil/events{/privacy}",
"organizations_url": "https://api.github.com/users/kevwil/orgs",
"url": "https://api.github.com/users/kevwil",
"gists_url": "https://api.github.com/users/kevwil/gists{/gist_id}",
"html_url": "https://github.com/kevwil",
"subscriptions_url": "https://api.github.com/users/kevwil/subscriptions",
"avatar_url": "https://avatars1.githubusercontent.com/u/35?v=3",
"repos_url": "https://api.github.com/users/kevwil/repos",
"received_events_url": "https://api.github.com/users/kevwil/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/kevwil/starred{/owner}{/repo}",
"site_admin": false,
"login": "kevwil",
"type": "User",
"id": 35,
"followers_url": "https://api.github.com/users/kevwil/followers"
},
{
"following_url": "https://api.github.com/users/KirinDave/following{/other_user}",
"events_url": "https://api.github.com/users/KirinDave/events{/privacy}",
"organizations_url": "https://api.github.com/users/KirinDave/orgs",
"url": "https://api.github.com/users/KirinDave",
"gists_url": "https://api.github.com/users/KirinDave/gists{/gist_id}",
"html_url": "https://github.com/KirinDave",
"subscriptions_url": "https://api.github.com/users/KirinDave/subscriptions",
"avatar_url": "https://avatars1.githubusercontent.com/u/36?v=3",
"repos_url": "https://api.github.com/users/KirinDave/repos",
"received_events_url": "https://api.github.com/users/KirinDave/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/KirinDave/starred{/owner}{/repo}",
"site_admin": false,
"login": "KirinDave",
"type": "User",
"id": 36,
"followers_url": "https://api.github.com/users/KirinDave/followers"
},
{
"following_url": "https://api.github.com/users/jamesgolick/following{/other_user}",
"events_url": "https://api.github.com/users/jamesgolick/events{/privacy}",
"organizations_url": "https://api.github.com/users/jamesgolick/orgs",
"url": "https://api.github.com/users/jamesgolick",
"gists_url": "https://api.github.com/users/jamesgolick/gists{/gist_id}",
"html_url": "https://github.com/jamesgolick",
"subscriptions_url": "https://api.github.com/users/jamesgolick/subscriptions",
"avatar_url": "https://avatars1.githubusercontent.com/u/37?v=3",
"repos_url": "https://api.github.com/users/jamesgolick/repos",
"received_events_url": "https://api.github.com/users/jamesgolick/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/jamesgolick/starred{/owner}{/repo}",
"site_admin": false,
"login": "jamesgolick",
"type": "User",
"id": 37,
"followers_url": "https://api.github.com/users/jamesgolick/followers"
},
{
"following_url": "https://api.github.com/users/atmos/following{/other_user}",
"events_url": "https://api.github.com/users/atmos/events{/privacy}",
"organizations_url": "https://api.github.com/users/atmos/orgs",
"url": "https://api.github.com/users/atmos",
"gists_url": "https://api.github.com/users/atmos/gists{/gist_id}",
"html_url": "https://github.com/atmos",
"subscriptions_url": "https://api.github.com/users/atmos/subscriptions",
"avatar_url": "https://avatars0.githubusercontent.com/u/38?v=3",
"repos_url": "https://api.github.com/users/atmos/repos",
"received_events_url": "https://api.github.com/users/atmos/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/atmos/starred{/owner}{/repo}",
"site_admin": false,
"login": "atmos",
"type": "User",
"id": 38,
"followers_url": "https://api.github.com/users/atmos/followers"
},
{
"following_url": "https://api.github.com/users/errfree/following{/other_user}",
"events_url": "https://api.github.com/users/errfree/events{/privacy}",
"organizations_url": "https://api.github.com/users/errfree/orgs",
"url": "https://api.github.com/users/errfree",
"gists_url": "https://api.github.com/users/errfree/gists{/gist_id}",
"html_url": "https://github.com/errfree",
"subscriptions_url": "https://api.github.com/users/errfree/subscriptions",
"avatar_url": "https://avatars1.githubusercontent.com/u/44?v=3",
"repos_url": "https://api.github.com/users/errfree/repos",
"received_events_url": "https://api.github.com/users/errfree/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/errfree/starred{/owner}{/repo}",
"site_admin": false,
"login": "errfree",
"type": "Organization",
"id": 44,
"followers_url": "https://api.github.com/users/errfree/followers"
},
{
"following_url": "https://api.github.com/users/mojodna/following{/other_user}",
"events_url": "https://api.github.com/users/mojodna/events{/privacy}",
"organizations_url": "https://api.github.com/users/mojodna/orgs",
"url": "https://api.github.com/users/mojodna",
"gists_url": "https://api.github.com/users/mojodna/gists{/gist_id}",
"html_url": "https://github.com/mojodna",
"subscriptions_url": "https://api.github.com/users/mojodna/subscriptions",
"avatar_url": "https://avatars1.githubusercontent.com/u/45?v=3",
"repos_url": "https://api.github.com/users/mojodna/repos",
"received_events_url": "https://api.github.com/users/mojodna/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/mojodna/starred{/owner}{/repo}",
"site_admin": false,
"login": "mojodna",
"type": "User",
"id": 45,
"followers_url": "https://api.github.com/users/mojodna/followers"
},
{
"following_url": "https://api.github.com/users/bmizerany/following{/other_user}",
"events_url": "https://api.github.com/users/bmizerany/events{/privacy}",
"organizations_url": "https://api.github.com/users/bmizerany/orgs",
"url": "https://api.github.com/users/bmizerany",
"gists_url": "https://api.github.com/users/bmizerany/gists{/gist_id}",
"html_url": "https://github.com/bmizerany",
"subscriptions_url": "https://api.github.com/users/bmizerany/subscriptions",
"avatar_url": "https://avatars1.githubusercontent.com/u/46?v=3",
"repos_url": "https://api.github.com/users/bmizerany/repos",
"received_events_url": "https://api.github.com/users/bmizerany/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/bmizerany/starred{/owner}{/repo}",
"site_admin": false,
"login": "bmizerany",
"type": "User",
"id": 46,
"followers_url": "https://api.github.com/users/bmizerany/followers"
},
{
"following_url": "https://api.github.com/users/jnewland/following{/other_user}",
"events_url": "https://api.github.com/users/jnewland/events{/privacy}",
"organizations_url": "https://api.github.com/users/jnewland/orgs",
"url": "https://api.github.com/users/jnewland",
"gists_url": "https://api.github.com/users/jnewland/gists{/gist_id}",
"html_url": "https://github.com/jnewland",
"subscriptions_url": "https://api.github.com/users/jnewland/subscriptions",
"avatar_url": "https://avatars1.githubusercontent.com/u/47?v=3",
"repos_url": "https://api.github.com/users/jnewland/repos",
"received_events_url": "https://api.github.com/users/jnewland/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/jnewland/starred{/owner}{/repo}",
"site_admin": true,
"login": "jnewland",
"type": "User",
"id": 47,
"followers_url": "https://api.github.com/users/jnewland/followers"
},
{
"following_url": "https://api.github.com/users/joshknowles/following{/other_user}",
"events_url": "https://api.github.com/users/joshknowles/events{/privacy}",
"organizations_url": "https://api.github.com/users/joshknowles/orgs",
"url": "https://api.github.com/users/joshknowles",
"gists_url": "https://api.github.com/users/joshknowles/gists{/gist_id}",
"html_url": "https://github.com/joshknowles",
"subscriptions_url": "https://api.github.com/users/joshknowles/subscriptions",
"avatar_url": "https://avatars0.githubusercontent.com/u/48?v=3",
"repos_url": "https://api.github.com/users/joshknowles/repos",
"received_events_url": "https://api.github.com/users/joshknowles/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/joshknowles/starred{/owner}{/repo}",
"site_admin": false,
"login": "joshknowles",
"type": "User",
"id": 48,
"followers_url": "https://api.github.com/users/joshknowles/followers"
},
{
"following_url": "https://api.github.com/users/hornbeck/following{/other_user}",
"events_url": "https://api.github.com/users/hornbeck/events{/privacy}",
"organizations_url": "https://api.github.com/users/hornbeck/orgs",
"url": "https://api.github.com/users/hornbeck",
"gists_url": "https://api.github.com/users/hornbeck/gists{/gist_id}",
"html_url": "https://github.com/hornbeck",
"subscriptions_url": "https://api.github.com/users/hornbeck/subscriptions",
"avatar_url": "https://avatars0.githubusercontent.com/u/49?v=3",
"repos_url": "https://api.github.com/users/hornbeck/repos",
"received_events_url": "https://api.github.com/users/hornbeck/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/hornbeck/starred{/owner}{/repo}",
"site_admin": false,
"login": "hornbeck",
"type": "User",
"id": 49,
"followers_url": "https://api.github.com/users/hornbeck/followers"
},
{
"following_url": "https://api.github.com/users/jwhitmire/following{/other_user}",
"events_url": "https://api.github.com/users/jwhitmire/events{/privacy}",
"organizations_url": "https://api.github.com/users/jwhitmire/orgs",
"url": "https://api.github.com/users/jwhitmire",
"gists_url": "https://api.github.com/users/jwhitmire/gists{/gist_id}",
"html_url": "https://github.com/jwhitmire",
"subscriptions_url": "https://api.github.com/users/jwhitmire/subscriptions",
"avatar_url": "https://avatars0.githubusercontent.com/u/50?v=3",
"repos_url": "https://api.github.com/users/jwhitmire/repos",
"received_events_url": "https://api.github.com/users/jwhitmire/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/jwhitmire/starred{/owner}{/repo}",
"site_admin": false,
"login": "jwhitmire",
"type": "User",
"id": 50,
"followers_url": "https://api.github.com/users/jwhitmire/followers"
},
{
"following_url": "https://api.github.com/users/elbowdonkey/following{/other_user}",
"events_url": "https://api.github.com/users/elbowdonkey/events{/privacy}",
"organizations_url": "https://api.github.com/users/elbowdonkey/orgs",
"url": "https://api.github.com/users/elbowdonkey",
"gists_url": "https://api.github.com/users/elbowdonkey/gists{/gist_id}",
"html_url": "https://github.com/elbowdonkey",
"subscriptions_url": "https://api.github.com/users/elbowdonkey/subscriptions",
"avatar_url": "https://avatars0.githubusercontent.com/u/51?v=3",
"repos_url": "https://api.github.com/users/elbowdonkey/repos",
"received_events_url": "https://api.github.com/users/elbowdonkey/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/elbowdonkey/starred{/owner}{/repo}",
"site_admin": false,
"login": "elbowdonkey",
"type": "User",
"id": 51,
"followers_url": "https://api.github.com/users/elbowdonkey/followers"
},
{
"following_url": "https://api.github.com/users/reinh/following{/other_user}",
"events_url": "https://api.github.com/users/reinh/events{/privacy}",
"organizations_url": "https://api.github.com/users/reinh/orgs",
"url": "https://api.github.com/users/reinh",
"gists_url": "https://api.github.com/users/reinh/gists{/gist_id}",
"html_url": "https://github.com/reinh",
"subscriptions_url": "https://api.github.com/users/reinh/subscriptions",
"avatar_url": "https://avatars0.githubusercontent.com/u/52?v=3",
"repos_url": "https://api.github.com/users/reinh/repos",
"received_events_url": "https://api.github.com/users/reinh/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/reinh/starred{/owner}{/repo}",
"site_admin": false,
"login": "reinh",
"type": "User",
"id": 52,
"followers_url": "https://api.github.com/users/reinh/followers"
},
{
"following_url": "https://api.github.com/users/knzconnor/following{/other_user}",
"events_url": "https://api.github.com/users/knzconnor/events{/privacy}",
"organizations_url": "https://api.github.com/users/knzconnor/orgs",
"url": "https://api.github.com/users/knzconnor",
"gists_url": "https://api.github.com/users/knzconnor/gists{/gist_id}",
"html_url": "https://github.com/knzconnor",
"subscriptions_url": "https://api.github.com/users/knzconnor/subscriptions",
"avatar_url": "https://avatars0.githubusercontent.com/u/53?v=3",
"repos_url": "https://api.github.com/users/knzconnor/repos",
"received_events_url": "https://api.github.com/users/knzconnor/received_events",
"gravatar_id": "",
"starred_url": "https://api.github.com/users/knzconnor/starred{/owner}{/repo}",
"site_admin": false,
"login": "knzconnor",
"type": "User",
"id": 53,
"followers_url": "https://api.github.com/users/knzconnor/followers"
}
]
{'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0'}
https://api.github.com/users?since=11
使用patch修改数据
def json_request():
response=requests.patch(build_uri('user'),auth=('charlesQQ','mypass'),
json={'name':'charlesQQ11','email':'charles_ali@qq.com'})
print better_print(response.text)
print response.request.headers
print response.request.body
print response.status_code
{
"disk_usage": 14729,
"private_gists": 0,
"public_repos": 7,
"site_admin": false,
"subscriptions_url": "https://api.github.com/users/CharlesQQ/subscriptions",
"gravatar_id": "",
"hireable": null,
"id": 18431718,
"followers_url": "https://api.github.com/users/CharlesQQ/followers",
"following_url": "https://api.github.com/users/CharlesQQ/following{/other_user}",
"collaborators": 0,
"total_private_repos": 0,
"blog": null,
"followers": 0,
"location": null,
"type": "User",
"email": "charles_ali@qq.com",
"bio": null,
"gists_url": "https://api.github.com/users/CharlesQQ/gists{/gist_id}",
"owned_private_repos": 0,
"company": null,
"events_url": "https://api.github.com/users/CharlesQQ/events{/privacy}",
"html_url": "https://github.com/CharlesQQ",
"updated_at": "2017-04-08T14:36:42Z",
"plan": {
"collaborators": 0,
"name": "free",
"private_repos": 0,
"space": 976562499
},
"received_events_url": "https://api.github.com/users/CharlesQQ/received_events",
"starred_url": "https://api.github.com/users/CharlesQQ/starred{/owner}{/repo}",
"public_gists": 0,
"name": "charlesQQ11",
"organizations_url": "https://api.github.com/users/CharlesQQ/orgs",
"url": "https://api.github.com/users/CharlesQQ",
"created_at": "2016-04-13T00:58:55Z",
"avatar_url": "https://avatars2.githubusercontent.com/u/18431718?v=3",
"repos_url": "https://api.github.com/users/CharlesQQ/repos",
"following": 8,
"login": "CharlesQQ",
"two_factor_authentication": false
}
{'Content-Length': '54', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Authorization': 'Basic Y2hhcmxlc1FROnN0YXJ0MzMzMzM='}
{"name": "charlesQQ11", "email": "charles_ali@qq.com"}
200
使用post增加数据
def json_request():
#response=requests.patch(build_uri('user'),auth=('charlesQQ','mypass'),
# json={'name':'charlesQQ11','email':'charles_ali@qq.com'})
response=requests.post(build_uri('user/emails'),auth=('charlesQQ','start33333'),json=['qq_c1231@163.com'])
print better_print(response.text)
print response.request.headers
print response.request.body
print response.status_code
六、请求异常处理
一般在我们调用第三方服务的时候，难以避免的是第三方服务调用出现异常，这种时候，就需要使用异常处理，来保证程序可以正常运行;
requests的异常处理，需要使用其exceptions类来实现;
将超时时间设置为0.1秒，来触发超时异常
import requests
from requests import exceptions
def timeout_request():
try:
response=requests.get(build_uri('user/emails'),timeout=0.1)
response.raise_for_status()
except exceptions.Timeout as e:
print e.message
except exceptions.HTTPError as e:
print e.message
else:
print response.text
print response.status_code
调用结果:
HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /user/emails (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003148CF8>, 'Connection to api.github.com timed out. (connect timeout=0.1)'))
将超时时间设置为10s，触发401错误
def timeout_request():
try:
response=requests.get(build_uri('user/emails'),timeout=10)
response.raise_for_status()
except exceptions.Timeout as e:
print e.message
except exceptions.HTTPError as e:
print e.message
else:
print response.text
print response.status_code
调用结果:
401 Client Error: Unauthorized for url: https://api.github.com/user/emails
七、自定义request
requests通过urlib3实现可以hold住整个socket，通过session实现;
发送的request包，包含发送的body，headers，auth，proxy，timeout，verify等，回复消息response为text,json等;
有时间去研究源码吧
def hard_requests():
from requests import Request,Session
s=Session()
headers={'User-Agent':'fake1.3.4'}
req=Request('GET',build_uri('user/emails'),auth=('myuser','mypass'),headers=headers)
prepared=req.prepare()
print prepared.body
print prepared.headers
resp=s.send(prepared,timeout=5)
print resp.status_code
print resp.request.headers
print resp.text
结果:
None
{'Authorization': 'Basic Y2hhcmxlc1FROnN0YXJ0MzMzMzM=', 'User-Agent': 'fake1.3.4'}
200
{'Authorization': 'Basic Y2hhcmxlc1FROnN0YXJ0MzMzMzM=', 'User-Agent': 'fake1.3.4'}
[{"email":"qq_c123@163.com","primary":false,"verified":false,"visibility":null},{"email":"charles_ali@qq.com","primary":true,"verified":true,"visibility":"public"},{"email":"qq_c1231@163.com","primary":false,"verified":false,"visibility":null}]
八、响应基本API
Response对象的API：
status_code:如下
400：Bad request,请求服务器端无法解析； 401:unauthorized,请求没有认证; 403：forbidden； 404:Not found；
500：内部错误;
reason:
headers:
url:
history:
elapsed:
request:
encoding:
raw:读取原始的对象
content:string类型:
text:unicode;
json:
下面分别举例介绍
status_code和reason
import requests
response = requests.get('https://api.github.com')
>>> response.status_code
200
>>> response.reason
'OK'
响应headers
>>> response.headers
{'X-XSS-Protection': '1; mode=block', 'Content-Security-Policy': "default-src 'none'", 'Access-Control-Expose-Headers': 'ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval', 'Transfer-Encoding': 'chunked', 'Access-Control-Allow-Origin': '*', 'X-Frame-Options': 'deny', 'Status': '200 OK', 'X-Served-By': 'eef8b8685a106934dcbb4b7c59fba0bf', 'X-GitHub-Request-Id': '059D:17F65:2FAC88D:3DE5D57:58EA0875', 'ETag': 'W/"7dc470913f1fe9bb6c7355b50a0737bc"', 'Date': 'Sun, 09 Apr 2017 10:09:57 GMT', 'X-RateLimit-Remaining': '38', 'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload', 'Server': 'GitHub.com', 'X-GitHub-Media-Type': 'github.v3; format=json', 'X-Content-Type-Options': 'nosniff', 'Content-Encoding': 'gzip', 'Vary': 'Accept, Accept-Encoding', 'X-RateLimit-Limit': '60', 'Cache-Control': 'public, max-age=60, s-maxage=60', 'Content-Type': 'application/json; charset=utf-8', 'X-RateLimit-Reset': '1491735173'}
响应url
>>> response.url u'https://api.github.com/'
响应history
>>> response.history #如果没有重定向，默认为空
[]
>>> response = requests.get('http://api.github.com') #一般访问http,会跳转到https
>>> response.history
[<Response [301]>]
响应时间
>>> response.elapsed datetime.timedelta(0, 3, 158723)
可以调用的方法
>>> dir(response) ['__attrs__', '__bool__', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getstate__', '__hash__', '__init__', '__iter__', '__module__', '__new__', '__nonzero__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_content', '_content_consumed', 'apparent_encoding', 'close', 'connection', 'content', 'cookies', 'elapsed', 'encoding', 'headers', 'history', 'is_permanent_redirect', 'is_redirect', 'iter_content', 'iter_lines', 'json', 'links', 'ok', 'raise_for_status', 'raw', 'reason', 'request', 'status_code', 'text', 'url']
请求对象和请求头
>>> response.request
<PreparedRequest [GET]>
>>> response.request.headers
{'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.el7.x86_64'}
请求的编码
>>> response.encoding 'utf-8'
响应的内容，content为str类型
>>> response.raw.read(10)
''
>>>
>>>
>>> response.content
'{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'
响应内容,text为unicode类型
>>> response.text
u'{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'
响应内容，为json对象
>>> response.json(
... )
{u'issues_url': u'https://api.github.com/issues', u'current_user_repositories_url': u'https://api.github.com/user/repos{?type,page,per_page,sort}', u'rate_limit_url': u'https://api.github.com/rate_limit', u'repository_search_url': u'https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}', u'user_organizations_url': u'https://api.github.com/user/orgs', u'commit_search_url': u'https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}', u'repository_url': u'https://api.github.com/repos/{owner}/{repo}', u'emojis_url': u'https://api.github.com/emojis', u'hub_url': u'https://api.github.com/hub', u'keys_url': u'https://api.github.com/user/keys', u'following_url': u'https://api.github.com/user/following{/target}', u'emails_url': u'https://api.github.com/user/emails', u'authorizations_url': u'https://api.github.com/authorizations', u'code_search_url': u'https://api.github.com/search/code?q={query}{&page,per_page,sort,order}', u'followers_url': u'https://api.github.com/user/followers', u'public_gists_url': u'https://api.github.com/gists/public', u'organization_url': u'https://api.github.com/orgs/{org}', u'gists_url': u'https://api.github.com/gists{/gist_id}', u'feeds_url': u'https://api.github.com/feeds', u'user_search_url': u'https://api.github.com/search/users?q={query}{&page,per_page,sort,order}', u'user_url': u'https://api.github.com/users/{user}', u'events_url': u'https://api.github.com/events', u'organization_repositories_url': u'https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}', u'current_user_url': u'https://api.github.com/user', u'issue_search_url': u'https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}', u'notifications_url': u'https://api.github.com/notifications', u'starred_url': u'https://api.github.com/user/starred{/owner}{/repo}', u'starred_gists_url': u'https://api.github.com/gists/starred', u'current_user_authorizations_html_url': u'https://github.com/settings/connections/applications{/client_id}', u'user_repositories_url': u'https://api.github.com/users/{user}/repos{?type,page,per_page,sort}', u'team_url': u'https://api.github.com/teams'}
>>> response.json()['team_url'] #访问json的对象
u'https://api.github.com/teams'
九、下载图片/文件
图片/文件下载过程
浏览器模拟-->构建request(调试策略)-->读取流data-->打开文件，存入数据
有一些返回的context的内容是没有办法读取的，只有通过流的方式保存到文件中
import requests
def download_image():
"""
demo:下载图片
:return:
"""
url= "http://img0.tech2ipo.com/upload/img/article/2015/06/1434420238601.png"
headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'}
response=requests.get(url,headers=headers,stream=True)
with open('demo.png','wb') as fd:
for chunk in response.iter_content(128):
fd.write(chunk)
print response.status_code
# print response.reason
# print response.content
使用上下午文管理的方式，读物数据流，写入文件，最后关闭流
def download_image_improved():
"""
demo:下载图片
:return:
"""
#伪造headers信息
headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'}
#限定url
url= "http://img0.tech2ipo.com/upload/img/article/2015/06/1434420238601.png"
response=requests.get(url,headers=headers,stream=True)
from contextlib import closing
with closing(requests.get(url,headers=headers,stream=True)) as response:
#打开文件
with open('demo.png','wb') as fd:
#每128写入一次
for chunk in response.iter_content(128):
fd.write(chunk)
十、事件钩子(Event hooks)
类似于js中的回调函数；
IO请求-->response-->回调函数
说白了，就是异步请求，request发出之后，直接返回，剩余的IO请求，由回调函数处理;
异步处理demo
import requests
def get_key_info(response,*args,**kwargs):
"""
回调函数
:return:
"""
print response.headers['Content-Type']
def main():
"""
主程序
:return:
"""
requests.get('https://api.github.com',hooks=dict(response=get_key_info))
main()
执行结果:
application/json; charset=utf-8
十一、HTTP认证
1、http基本认证
requests a protected resource-->requests a username:password-->sends username:password-->returns requests resource
import requests
BASE_URL='https://api.github.com'
def construct_url(end_point):
return '/'.join([BASE_URL,end_point])
def basic_auth():
"""
基本认证
:return:
"""
response=requests.get(construct_url('user'),auth=('imoocdeo','imoocdeo123'))
print response.text
print response.request.headers
basic_auth()
打印结果:
{"message":"Bad credentials","documentation_url":"https://developer.github.com/v3"}
{'Authorization': 'Basic aW1vb2NkZW86aW1vb2NkZW8xMjM=', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0'}
这里字符串 aW1vb2NkZW86aW1vb2NkZW8xMjM=是使用base64编码的结果
在终端打开bpython(pip install bpython安装)
>>> import base64
>>>
>>> base64.b64decode('aW1vb2NkZW86aW1vb2NkZW8xMjM=')
'imoocdeo:imoocdeo123' #是用户名和密码拼凑起来的,但是这样可能不安全
2、OAUTH认证
https://www.codewars.com/ 使用的登录认证方式
APP---github
github生成access_token: https://github.com/settings/tokens-->Generate new token-->选择该token拥有的权限
def basic_oauth():
headers={'Authorization':'token 51684c8452ff83556f1a6705d2d85d7f68cd5ca7'}
#user/emails,通过请求头把token带过去
response=requests.get(construct_url('user/emails'),headers=headers)
print response.request.headers
print response.text
print response.status_code
basic_oauth()
打印结果:
{'Authorization': 'token 51684c8452ff83556f1a6705d2d85d7f68cd5ca7', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0'}
[{"email":"qq_c123@163.com","primary":false,"verified":false,"visibility":null},{"email":"charles_ali@qq.com","primary":true,"verified":true,"visibility":"public"},{"email":"qq_c1231@163.com","primary":false,"verified":false,"visibility":null}]
200
使用python自带的语法糖类实现上述内容,__call__可以将实例化的对象当做类进行实例化调用,即将实例化的对象在加()，就可以调用__call__函数,f()();
from requests.auth import AuthBase
class GithubAuth(AuthBase):
def __init__(self,token):
self.token=token
def __call__(self, r):
#request 加headers
r.headers['Authorization']=' '.join(['token',self.token])
print r
return r
def oauth_advaced():
auth=GithubAuth('51684c8452ff83556f1a6705d2d85d7f68cd5ca7')
response=requests.get(construct_url('user/emails'),auth=auth)
print response.text
oauth_advaced()
调用结果:
<PreparedRequest [GET]>
[{"email":"qq_c123@163.com","primary":false,"verified":false,"visibility":null},{"email":"charles_ali@qq.com","primary":true,"verified":true,"visibility":"public"},{"email":"qq_c1231@163.com","primary":false,"verified":false,"visibility":null}]
十二、requests库的proxy代理
import requests
proxies={'http':'sock5://127.0.0.1:1080','https':'sock5://127.0.0.1:1080'} #设置代理地址
url='https://www.facebook.com'
response=requests.get(url,proxies=proxies,timeout=10)
十三、Session和cookie
cookie:
浏览器请求(无cookie)-->服务器-->HTTP响应(响应头有set cookie,设置有效期，权限范围等等)-->解析cookie保存本地-->HTTP请求(携带cookie)--->解析cookie识别信息--HTTP响应
缺点:cookie是在浏览器端的，每次请求，都要携带cookie，消耗带宽;cookie可以伪造,不安全;
session:
浏览器HTTP请求-->服务器存储session-->HTTP响应(set cookie-session-id 叫做session id的cookie)-->解析cookie保存本地(这个cookie非常小,仅含有session id)-->解析session id(到存储中查，该session id对于的用户信息，权限信息是什么)-->HTTP响应