requests库

一、http协议原理

http是无状态的应用层协议

二、urllib、urllib2、requests

在python2中，有urllib和urlib2两个包，python3全变成urllib了

urllib和urllib2是相互独立的模块,比如，urllib不能伪造user agent，但是请求参数的构建，需要使用到urllib,所有经常是两者配合使用的;

requests使用了urlib3(多次请求重复使用一个socket)

三、环境准备

http://httpbin.org/ 

如果将该站点搭建在本地，使用pip安装就可以了

pip install gunicorn httpbin

启动:

gunicorn httpbin:app --bind 0.0.0.0:8000

发送一个http的get请求，获取本机的IP地址

import urllib2
import urllib

URL_IP='http://192.168.74.20:8000/ip'

def use_simple_urllib2():
    response=urllib2.urlopen(URL_IP)
    print ">>>>Response Headers:"
    print response.info()
    print ''.join([line for line in response.readlines()])

if __name__ == '__main__':
    print ">>>Use Simple urllib2"
    use_simple_urllib2()

结果为:
>>>Use Simple urllib2
>>>>Response Headers:
Server: gunicorn/19.7.1
Date: Sat, 08 Apr 2017 13:06:18 GMT
Connection: close
Content-Type: application/json
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Length: 31

{
  "origin": "192.168.74.1"
}

　　

使用urllib构建请求参数,使用urllib2发送请求

def use_params_urllib2():
    """
    构建请求参数
    :return:
    """
    params = urllib.urlencode({'param1':'hello','param2':'world'})
    print 'Request Params:'
    print params
    #发送请求
    response=urllib2.urlopen('?'.join([URL_GET,'%s'])%params)
    #处理响应
    print ">>>>Response Headers:"
    print response.info()
    print ">>>Status code"
    print response.getcode()
    print ''.join([line for line in response.readlines()])
if __name__ == '__main__':
    print ">>>Use Simple urllib2"
    use_simple_urllib2()
    print ">>>Use params urllib2"
    use_params_urllib2()

结果:
>>>Use params urllib2
Request Params:
param2=world&param1=hello
>>>>Response Headers:
Server: gunicorn/19.7.1
Date: Sat, 08 Apr 2017 13:22:05 GMT
Connection: close
Content-Type: application/json
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Length: 322

>>>Status code
200
{
  "args": {
    "param1": "hello", 
    "param2": "world"
  }, 
  "headers": {
    "Accept-Encoding": "identity", 
    "Connection": "close", 
    "Host": "192.168.74.20:8000", 
    "User-Agent": "Python-urllib/2.7"
  }, 
  "origin": "192.168.74.1", 
  "url": "http://192.168.74.20:8000/get?param2=world&param1=hello"
}

　

使用requests模块实现上述功能

#!/usr/bin/env python
# _*_ coding:utf-8 _*_
__author__ = 'Charles Chang'

import requests
import urllib2
import urllib

URL_IP='http://192.168.74.20:8000/ip'
URL_GET='http://192.168.74.20:8000/get'

def use_simple_requests():
    response=requests.get(URL_IP)
    print ">>>>Response Headers:"
    print response.headers
    print '>>>>Response Body:'
    print response.text

def use_params_resuqets():
    """
    构建请求参数
    :return:
    """
    params = {'param1':'hello','param2':'world'}
    print 'Request Params:'
    print params
    #发送请求
    response=requests.get(URL_IP,params=params)
    #处理响应
    print ">>>>Response Headers:"
    print response.headers
    print ">>>Status code"
    print response.status_code
    print response.reason
    print ">>>>Response Body:"
    print response.json()

if __name__ == '__main__':
    print ">>>Use Simple urllib2"
    use_simple_requests()
    print
    print ">>>Use params requests"
    use_params_resuqets()

结果：
>>>Use Simple urllib2
>>>>Response Headers:
{'Content-Length': '31', 'Server': 'gunicorn/19.7.1', 'Connection': 'close', 'Access-Control-Allow-Credentials': 'true', 'Date': 'Sat, 08 Apr 2017 14:29:38 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json'}
>>>>Response Body:
{
  "origin": "192.168.74.1"
}


>>>Use params requests
Request Params:
{'param2': 'world', 'param1': 'hello'}
>>>>Response Headers:
{'Content-Length': '31', 'Server': 'gunicorn/19.7.1', 'Connection': 'close', 'Access-Control-Allow-Credentials': 'true', 'Date': 'Sat, 08 Apr 2017 14:29:38 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json'}
>>>Status code
200
OK
>>>>Response Body:
{u'origin': u'192.168.74.1'}

　　

　

四、github api，requests 请求方法

https://developer.github.com/v3/

GET:查看资源

POST:增加资源

PUT：修改资源

DELETE：删除资源

HEAD:查看响应头

OPTIONS:查看可用请求方法

PATCH:使用少量的json格式的数据去更新数据的方式;

Verb	Description
HEAD	Can be issued against any resource to get just the HTTP header info.
GET	Used for retrieving resources.
POST	Used for creating resources.
PATCH	Used for updating resources with partial JSON data. For instance, an Issue resource hastitle and body attributes. A PATCH request may accept one or more of the attributes to update the resource. PATCH is a relatively new and uncommon HTTP verb, so resource endpoints also accept POST requests.
PUT	Used for replacing resources or collections. For PUT requests with no body attribute, be sure to set the Content-Length header to zero.
DELETE	Used for deleting resources.

requests请求方法调用github api

import json
import requests

URL='https://api.github.com'

def build_uri(endpoint):
    return '/'.join([URL,endpoint])

def better_print(json_str):
    return json.dumps(json.loads(json_str),indent=4)

def request_method():
    response=requests.get(build_uri('users/myuser'),auth=('myuser','mypassword'))
    print response.text
    print better_print(response.text)

if __name__ == '__main__':
    request_method()

结果:
{"login":"CharlesQQ","id":18431718,"avatar_url":"https://avatars2.githubusercontent.com/u/18431718?v=3","gravatar_id":"","url":"https://api.github.com/users/CharlesQQ","html_url":"https://github.com/CharlesQQ","followers_url":"https://api.github.com/users/CharlesQQ/followers","following_url":"https://api.github.com/users/CharlesQQ/following{/other_user}","gists_url":"https://api.github.com/users/CharlesQQ/gists{/gist_id}","starred_url":"https://api.github.com/users/CharlesQQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CharlesQQ/subscriptions","organizations_url":"https://api.github.com/users/CharlesQQ/orgs","repos_url":"https://api.github.com/users/CharlesQQ/repos","events_url":"https://api.github.com/users/CharlesQQ/events{/privacy}","received_events_url":"https://api.github.com/users/CharlesQQ/received_events","type":"User","site_admin":false,"name":null,"company":null,"blog":null,"location":null,"email":null,"hireable":null,"bio":null,"public_repos":7,"public_gists":0,"followers":0,"following":8,"created_at":"2016-04-13T00:58:55Z","updated_at":"2017-04-08T14:36:42Z","private_gists":0,"total_private_repos":0,"owned_private_repos":0,"disk_usage":14729,"collaborators":0,"two_factor_authentication":false,"plan":{"name":"free","space":976562499,"collaborators":0,"private_repos":0}}
{
    "disk_usage": 14729, 
    "private_gists": 0, 
    "public_repos": 7, 
    "site_admin": false, 
    "subscriptions_url": "https://api.github.com/users/CharlesQQ/subscriptions", 
    "gravatar_id": "", 
    "hireable": null, 
    "id": 18431718, 
    "followers_url": "https://api.github.com/users/CharlesQQ/followers", 
    "following_url": "https://api.github.com/users/CharlesQQ/following{/other_user}", 
    "collaborators": 0, 
    "total_private_repos": 0, 
    "blog": null, 
    "followers": 0, 
    "location": null, 
    "type": "User", 
    "email": null, 
    "bio": null, 
    "gists_url": "https://api.github.com/users/CharlesQQ/gists{/gist_id}", 
    "owned_private_repos": 0, 
    "company": null, 
    "events_url": "https://api.github.com/users/CharlesQQ/events{/privacy}", 
    "html_url": "https://github.com/CharlesQQ", 
    "updated_at": "2017-04-08T14:36:42Z", 
    "plan": {
        "collaborators": 0, 
        "name": "free", 
        "private_repos": 0, 
        "space": 976562499
    }, 
    "received_events_url": "https://api.github.com/users/CharlesQQ/received_events", 
    "starred_url": "https://api.github.com/users/CharlesQQ/starred{/owner}{/repo}", 
    "public_gists": 0, 
    "name": null, 
    "organizations_url": "https://api.github.com/users/CharlesQQ/orgs", 
    "url": "https://api.github.com/users/CharlesQQ", 
    "created_at": "2016-04-13T00:58:55Z", 
    "avatar_url": "https://avatars2.githubusercontent.com/u/18431718?v=3", 
    "repos_url": "https://api.github.com/users/CharlesQQ/repos", 
    "following": 8, 
    "login": "CharlesQQ", 
    "two_factor_authentication": false
}

五、带参数的请求

方式1:

search_product.html?cat=19283377&...

params: requests.get(url,params={'key1':''value1})

方式2：表单数据提交

Content-Type: application/x-www-form-urlencoded

requests.post(url,data={'key1':'value1','keys':'value2'})

方式3：json参数提交

Content-type:application/json

requests.post(url,json={'key1':'value1','key2':'value2'})

使用params获取user信息

def params_requests():
    response=requests.get(build_uri('users'),params={'since':11})
    print better_print(response.text)
    print response.request.headers
    print response.url
if __name__ == '__main__':
    params_requests()

结果：
    {
        "following_url": "https://api.github.com/users/tomtt/following{/other_user}", 
        "events_url": "https://api.github.com/users/tomtt/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/tomtt/orgs", 
        "url": "https://api.github.com/users/tomtt", 
        "gists_url": "https://api.github.com/users/tomtt/gists{/gist_id}", 
        "html_url": "https://github.com/tomtt", 
        "subscriptions_url": "https://api.github.com/users/tomtt/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/31?v=3", 
        "repos_url": "https://api.github.com/users/tomtt/repos", 
        "received_events_url": "https://api.github.com/users/tomtt/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/tomtt/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "tomtt", 
        "type": "User", 
        "id": 31, 
        "followers_url": "https://api.github.com/users/tomtt/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/railsjitsu/following{/other_user}", 
        "events_url": "https://api.github.com/users/railsjitsu/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/railsjitsu/orgs", 
        "url": "https://api.github.com/users/railsjitsu", 
        "gists_url": "https://api.github.com/users/railsjitsu/gists{/gist_id}", 
        "html_url": "https://github.com/railsjitsu", 
        "subscriptions_url": "https://api.github.com/users/railsjitsu/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/32?v=3", 
        "repos_url": "https://api.github.com/users/railsjitsu/repos", 
        "received_events_url": "https://api.github.com/users/railsjitsu/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/railsjitsu/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "railsjitsu", 
        "type": "User", 
        "id": 32, 
        "followers_url": "https://api.github.com/users/railsjitsu/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/nitay/following{/other_user}", 
        "events_url": "https://api.github.com/users/nitay/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/nitay/orgs", 
        "url": "https://api.github.com/users/nitay", 
        "gists_url": "https://api.github.com/users/nitay/gists{/gist_id}", 
        "html_url": "https://github.com/nitay", 
        "subscriptions_url": "https://api.github.com/users/nitay/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/34?v=3", 
        "repos_url": "https://api.github.com/users/nitay/repos", 
        "received_events_url": "https://api.github.com/users/nitay/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/nitay/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "nitay", 
        "type": "User", 
        "id": 34, 
        "followers_url": "https://api.github.com/users/nitay/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/kevwil/following{/other_user}", 
        "events_url": "https://api.github.com/users/kevwil/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/kevwil/orgs", 
        "url": "https://api.github.com/users/kevwil", 
        "gists_url": "https://api.github.com/users/kevwil/gists{/gist_id}", 
        "html_url": "https://github.com/kevwil", 
        "subscriptions_url": "https://api.github.com/users/kevwil/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/35?v=3", 
        "repos_url": "https://api.github.com/users/kevwil/repos", 
        "received_events_url": "https://api.github.com/users/kevwil/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/kevwil/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "kevwil", 
        "type": "User", 
        "id": 35, 
        "followers_url": "https://api.github.com/users/kevwil/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/KirinDave/following{/other_user}", 
        "events_url": "https://api.github.com/users/KirinDave/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/KirinDave/orgs", 
        "url": "https://api.github.com/users/KirinDave", 
        "gists_url": "https://api.github.com/users/KirinDave/gists{/gist_id}", 
        "html_url": "https://github.com/KirinDave", 
        "subscriptions_url": "https://api.github.com/users/KirinDave/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/36?v=3", 
        "repos_url": "https://api.github.com/users/KirinDave/repos", 
        "received_events_url": "https://api.github.com/users/KirinDave/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/KirinDave/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "KirinDave", 
        "type": "User", 
        "id": 36, 
        "followers_url": "https://api.github.com/users/KirinDave/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/jamesgolick/following{/other_user}", 
        "events_url": "https://api.github.com/users/jamesgolick/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/jamesgolick/orgs", 
        "url": "https://api.github.com/users/jamesgolick", 
        "gists_url": "https://api.github.com/users/jamesgolick/gists{/gist_id}", 
        "html_url": "https://github.com/jamesgolick", 
        "subscriptions_url": "https://api.github.com/users/jamesgolick/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/37?v=3", 
        "repos_url": "https://api.github.com/users/jamesgolick/repos", 
        "received_events_url": "https://api.github.com/users/jamesgolick/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/jamesgolick/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "jamesgolick", 
        "type": "User", 
        "id": 37, 
        "followers_url": "https://api.github.com/users/jamesgolick/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/atmos/following{/other_user}", 
        "events_url": "https://api.github.com/users/atmos/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/atmos/orgs", 
        "url": "https://api.github.com/users/atmos", 
        "gists_url": "https://api.github.com/users/atmos/gists{/gist_id}", 
        "html_url": "https://github.com/atmos", 
        "subscriptions_url": "https://api.github.com/users/atmos/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/38?v=3", 
        "repos_url": "https://api.github.com/users/atmos/repos", 
        "received_events_url": "https://api.github.com/users/atmos/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/atmos/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "atmos", 
        "type": "User", 
        "id": 38, 
        "followers_url": "https://api.github.com/users/atmos/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/errfree/following{/other_user}", 
        "events_url": "https://api.github.com/users/errfree/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/errfree/orgs", 
        "url": "https://api.github.com/users/errfree", 
        "gists_url": "https://api.github.com/users/errfree/gists{/gist_id}", 
        "html_url": "https://github.com/errfree", 
        "subscriptions_url": "https://api.github.com/users/errfree/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/44?v=3", 
        "repos_url": "https://api.github.com/users/errfree/repos", 
        "received_events_url": "https://api.github.com/users/errfree/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/errfree/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "errfree", 
        "type": "Organization", 
        "id": 44, 
        "followers_url": "https://api.github.com/users/errfree/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/mojodna/following{/other_user}", 
        "events_url": "https://api.github.com/users/mojodna/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/mojodna/orgs", 
        "url": "https://api.github.com/users/mojodna", 
        "gists_url": "https://api.github.com/users/mojodna/gists{/gist_id}", 
        "html_url": "https://github.com/mojodna", 
        "subscriptions_url": "https://api.github.com/users/mojodna/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/45?v=3", 
        "repos_url": "https://api.github.com/users/mojodna/repos", 
        "received_events_url": "https://api.github.com/users/mojodna/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/mojodna/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "mojodna", 
        "type": "User", 
        "id": 45, 
        "followers_url": "https://api.github.com/users/mojodna/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/bmizerany/following{/other_user}", 
        "events_url": "https://api.github.com/users/bmizerany/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/bmizerany/orgs", 
        "url": "https://api.github.com/users/bmizerany", 
        "gists_url": "https://api.github.com/users/bmizerany/gists{/gist_id}", 
        "html_url": "https://github.com/bmizerany", 
        "subscriptions_url": "https://api.github.com/users/bmizerany/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/46?v=3", 
        "repos_url": "https://api.github.com/users/bmizerany/repos", 
        "received_events_url": "https://api.github.com/users/bmizerany/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/bmizerany/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "bmizerany", 
        "type": "User", 
        "id": 46, 
        "followers_url": "https://api.github.com/users/bmizerany/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/jnewland/following{/other_user}", 
        "events_url": "https://api.github.com/users/jnewland/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/jnewland/orgs", 
        "url": "https://api.github.com/users/jnewland", 
        "gists_url": "https://api.github.com/users/jnewland/gists{/gist_id}", 
        "html_url": "https://github.com/jnewland", 
        "subscriptions_url": "https://api.github.com/users/jnewland/subscriptions", 
        "avatar_url": "https://avatars1.githubusercontent.com/u/47?v=3", 
        "repos_url": "https://api.github.com/users/jnewland/repos", 
        "received_events_url": "https://api.github.com/users/jnewland/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/jnewland/starred{/owner}{/repo}", 
        "site_admin": true, 
        "login": "jnewland", 
        "type": "User", 
        "id": 47, 
        "followers_url": "https://api.github.com/users/jnewland/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/joshknowles/following{/other_user}", 
        "events_url": "https://api.github.com/users/joshknowles/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/joshknowles/orgs", 
        "url": "https://api.github.com/users/joshknowles", 
        "gists_url": "https://api.github.com/users/joshknowles/gists{/gist_id}", 
        "html_url": "https://github.com/joshknowles", 
        "subscriptions_url": "https://api.github.com/users/joshknowles/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/48?v=3", 
        "repos_url": "https://api.github.com/users/joshknowles/repos", 
        "received_events_url": "https://api.github.com/users/joshknowles/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/joshknowles/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "joshknowles", 
        "type": "User", 
        "id": 48, 
        "followers_url": "https://api.github.com/users/joshknowles/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/hornbeck/following{/other_user}", 
        "events_url": "https://api.github.com/users/hornbeck/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/hornbeck/orgs", 
        "url": "https://api.github.com/users/hornbeck", 
        "gists_url": "https://api.github.com/users/hornbeck/gists{/gist_id}", 
        "html_url": "https://github.com/hornbeck", 
        "subscriptions_url": "https://api.github.com/users/hornbeck/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/49?v=3", 
        "repos_url": "https://api.github.com/users/hornbeck/repos", 
        "received_events_url": "https://api.github.com/users/hornbeck/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/hornbeck/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "hornbeck", 
        "type": "User", 
        "id": 49, 
        "followers_url": "https://api.github.com/users/hornbeck/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/jwhitmire/following{/other_user}", 
        "events_url": "https://api.github.com/users/jwhitmire/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/jwhitmire/orgs", 
        "url": "https://api.github.com/users/jwhitmire", 
        "gists_url": "https://api.github.com/users/jwhitmire/gists{/gist_id}", 
        "html_url": "https://github.com/jwhitmire", 
        "subscriptions_url": "https://api.github.com/users/jwhitmire/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/50?v=3", 
        "repos_url": "https://api.github.com/users/jwhitmire/repos", 
        "received_events_url": "https://api.github.com/users/jwhitmire/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/jwhitmire/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "jwhitmire", 
        "type": "User", 
        "id": 50, 
        "followers_url": "https://api.github.com/users/jwhitmire/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/elbowdonkey/following{/other_user}", 
        "events_url": "https://api.github.com/users/elbowdonkey/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/elbowdonkey/orgs", 
        "url": "https://api.github.com/users/elbowdonkey", 
        "gists_url": "https://api.github.com/users/elbowdonkey/gists{/gist_id}", 
        "html_url": "https://github.com/elbowdonkey", 
        "subscriptions_url": "https://api.github.com/users/elbowdonkey/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/51?v=3", 
        "repos_url": "https://api.github.com/users/elbowdonkey/repos", 
        "received_events_url": "https://api.github.com/users/elbowdonkey/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/elbowdonkey/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "elbowdonkey", 
        "type": "User", 
        "id": 51, 
        "followers_url": "https://api.github.com/users/elbowdonkey/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/reinh/following{/other_user}", 
        "events_url": "https://api.github.com/users/reinh/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/reinh/orgs", 
        "url": "https://api.github.com/users/reinh", 
        "gists_url": "https://api.github.com/users/reinh/gists{/gist_id}", 
        "html_url": "https://github.com/reinh", 
        "subscriptions_url": "https://api.github.com/users/reinh/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/52?v=3", 
        "repos_url": "https://api.github.com/users/reinh/repos", 
        "received_events_url": "https://api.github.com/users/reinh/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/reinh/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "reinh", 
        "type": "User", 
        "id": 52, 
        "followers_url": "https://api.github.com/users/reinh/followers"
    }, 
    {
        "following_url": "https://api.github.com/users/knzconnor/following{/other_user}", 
        "events_url": "https://api.github.com/users/knzconnor/events{/privacy}", 
        "organizations_url": "https://api.github.com/users/knzconnor/orgs", 
        "url": "https://api.github.com/users/knzconnor", 
        "gists_url": "https://api.github.com/users/knzconnor/gists{/gist_id}", 
        "html_url": "https://github.com/knzconnor", 
        "subscriptions_url": "https://api.github.com/users/knzconnor/subscriptions", 
        "avatar_url": "https://avatars0.githubusercontent.com/u/53?v=3", 
        "repos_url": "https://api.github.com/users/knzconnor/repos", 
        "received_events_url": "https://api.github.com/users/knzconnor/received_events", 
        "gravatar_id": "", 
        "starred_url": "https://api.github.com/users/knzconnor/starred{/owner}{/repo}", 
        "site_admin": false, 
        "login": "knzconnor", 
        "type": "User", 
        "id": 53, 
        "followers_url": "https://api.github.com/users/knzconnor/followers"
    }
]
{'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0'}
https://api.github.com/users?since=11

　　

使用patch修改数据

def json_request():
    response=requests.patch(build_uri('user'),auth=('charlesQQ','mypass'),
                            json={'name':'charlesQQ11','email':'charles_ali@qq.com'})
    print better_print(response.text)
    print response.request.headers
    print response.request.body
    print response.status_code

{
    "disk_usage": 14729, 
    "private_gists": 0, 
    "public_repos": 7, 
    "site_admin": false, 
    "subscriptions_url": "https://api.github.com/users/CharlesQQ/subscriptions", 
    "gravatar_id": "", 
    "hireable": null, 
    "id": 18431718, 
    "followers_url": "https://api.github.com/users/CharlesQQ/followers", 
    "following_url": "https://api.github.com/users/CharlesQQ/following{/other_user}", 
    "collaborators": 0, 
    "total_private_repos": 0, 
    "blog": null, 
    "followers": 0, 
    "location": null, 
    "type": "User", 
    "email": "charles_ali@qq.com", 
    "bio": null, 
    "gists_url": "https://api.github.com/users/CharlesQQ/gists{/gist_id}", 
    "owned_private_repos": 0, 
    "company": null, 
    "events_url": "https://api.github.com/users/CharlesQQ/events{/privacy}", 
    "html_url": "https://github.com/CharlesQQ", 
    "updated_at": "2017-04-08T14:36:42Z", 
    "plan": {
        "collaborators": 0, 
        "name": "free", 
        "private_repos": 0, 
        "space": 976562499
    }, 
    "received_events_url": "https://api.github.com/users/CharlesQQ/received_events", 
    "starred_url": "https://api.github.com/users/CharlesQQ/starred{/owner}{/repo}", 
    "public_gists": 0, 
    "name": "charlesQQ11", 
    "organizations_url": "https://api.github.com/users/CharlesQQ/orgs", 
    "url": "https://api.github.com/users/CharlesQQ", 
    "created_at": "2016-04-13T00:58:55Z", 
    "avatar_url": "https://avatars2.githubusercontent.com/u/18431718?v=3", 
    "repos_url": "https://api.github.com/users/CharlesQQ/repos", 
    "following": 8, 
    "login": "CharlesQQ", 
    "two_factor_authentication": false
}
{'Content-Length': '54', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Authorization': 'Basic Y2hhcmxlc1FROnN0YXJ0MzMzMzM='}
{"name": "charlesQQ11", "email": "charles_ali@qq.com"}
200

　　

 使用post增加数据

def json_request():
    #response=requests.patch(build_uri('user'),auth=('charlesQQ','mypass'),
                           # json={'name':'charlesQQ11','email':'charles_ali@qq.com'})
    response=requests.post(build_uri('user/emails'),auth=('charlesQQ','start33333'),json=['qq_c1231@163.com'])
    print better_print(response.text)
    print response.request.headers
    print response.request.body
    print response.status_code

　　

六、请求异常处理

一般在我们调用第三方服务的时候，难以避免的是第三方服务调用出现异常，这种时候，就需要使用异常处理，来保证程序可以正常运行;

requests的异常处理，需要使用其exceptions类来实现;

将超时时间设置为0.1秒，来触发超时异常

import requests
from requests import exceptions
def timeout_request():
    try:
        response=requests.get(build_uri('user/emails'),timeout=0.1)
        response.raise_for_status()
    except exceptions.Timeout as e:
        print e.message
    except exceptions.HTTPError as e:
        print e.message
    else:
        print response.text
        print response.status_code

调用结果:
HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /user/emails (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003148CF8>, 'Connection to api.github.com timed out. (connect timeout=0.1)'))

将超时时间设置为10s，触发401错误

def timeout_request():
    try:
        response=requests.get(build_uri('user/emails'),timeout=10)
        response.raise_for_status()
    except exceptions.Timeout as e:
        print e.message
    except exceptions.HTTPError as e:
        print e.message
    else:
        print response.text
        print response.status_code

调用结果:
401 Client Error: Unauthorized for url: https://api.github.com/user/emails

七、自定义request

requests通过urlib3实现可以hold住整个socket，通过session实现;

发送的request包，包含发送的body，headers，auth，proxy，timeout，verify等，回复消息response为text,json等;

有时间去研究源码吧

def hard_requests():
    from requests import Request,Session
    s=Session()
    headers={'User-Agent':'fake1.3.4'}
    req=Request('GET',build_uri('user/emails'),auth=('myuser','mypass'),headers=headers)
    prepared=req.prepare()
    print prepared.body
    print prepared.headers

    resp=s.send(prepared,timeout=5)
    print resp.status_code
    print resp.request.headers
    print resp.text

结果:
None
{'Authorization': 'Basic Y2hhcmxlc1FROnN0YXJ0MzMzMzM=', 'User-Agent': 'fake1.3.4'}
200
{'Authorization': 'Basic Y2hhcmxlc1FROnN0YXJ0MzMzMzM=', 'User-Agent': 'fake1.3.4'}
[{"email":"qq_c123@163.com","primary":false,"verified":false,"visibility":null},{"email":"charles_ali@qq.com","primary":true,"verified":true,"visibility":"public"},{"email":"qq_c1231@163.com","primary":false,"verified":false,"visibility":null}]

　　

八、响应基本API

Response对象的API：

   status_code:如下

       400：Bad request,请求服务器端无法解析；   401:unauthorized,请求没有认证;  403：forbidden；  404:Not found；

       500：内部错误;

  reason:

  headers:

  url:

  history:

  elapsed:

   request:

   encoding:

  raw:读取原始的对象

  content:string类型:

   text:unicode;

   json:

  

下面分别举例介绍

status_code和reason

import requests
response = requests.get('https://api.github.com')
>>> response.status_code
200
>>> response.reason
'OK'

响应headers

>>> response.headers
{'X-XSS-Protection': '1; mode=block', 'Content-Security-Policy': "default-src 'none'", 'Access-Control-Expose-Headers': 'ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval', 'Transfer-Encoding': 'chunked', 'Access-Control-Allow-Origin': '*', 'X-Frame-Options': 'deny', 'Status': '200 OK', 'X-Served-By': 'eef8b8685a106934dcbb4b7c59fba0bf', 'X-GitHub-Request-Id': '059D:17F65:2FAC88D:3DE5D57:58EA0875', 'ETag': 'W/"7dc470913f1fe9bb6c7355b50a0737bc"', 'Date': 'Sun, 09 Apr 2017 10:09:57 GMT', 'X-RateLimit-Remaining': '38', 'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload', 'Server': 'GitHub.com', 'X-GitHub-Media-Type': 'github.v3; format=json', 'X-Content-Type-Options': 'nosniff', 'Content-Encoding': 'gzip', 'Vary': 'Accept, Accept-Encoding', 'X-RateLimit-Limit': '60', 'Cache-Control': 'public, max-age=60, s-maxage=60', 'Content-Type': 'application/json; charset=utf-8', 'X-RateLimit-Reset': '1491735173'}

响应url

>>> response.url
u'https://api.github.com/'

　　

响应history

>>> response.history    #如果没有重定向，默认为空
[]

>>> response = requests.get('http://api.github.com')    #一般访问http,会跳转到https
>>> response.history
[<Response [301]>]

响应时间

>>> response.elapsed
datetime.timedelta(0, 3, 158723)

可以调用的方法

>>> dir(response)
['__attrs__', '__bool__', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getstate__', '__hash__', '__init__', '__iter__', '__module__', '__new__', '__nonzero__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_content', '_content_consumed', 'apparent_encoding', 'close', 'connection', 'content', 'cookies', 'elapsed', 'encoding', 'headers', 'history', 'is_permanent_redirect', 'is_redirect', 'iter_content', 'iter_lines', 'json', 'links', 'ok', 'raise_for_status', 'raw', 'reason', 'request', 'status_code', 'text', 'url']

请求对象和请求头

>>> response.request
<PreparedRequest [GET]>
>>> response.request.headers
{'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.el7.x86_64'}

　　

请求的编码

>>> response.encoding
'utf-8'

响应的内容，content为str类型

>>> response.raw.read(10)
''
>>> 
>>> 
>>> response.content
'{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'

　　

响应内容,text为unicode类型

>>> response.text
u'{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'

　　

响应内容，为json对象

>>> response.json(
... )
{u'issues_url': u'https://api.github.com/issues', u'current_user_repositories_url': u'https://api.github.com/user/repos{?type,page,per_page,sort}', u'rate_limit_url': u'https://api.github.com/rate_limit', u'repository_search_url': u'https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}', u'user_organizations_url': u'https://api.github.com/user/orgs', u'commit_search_url': u'https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}', u'repository_url': u'https://api.github.com/repos/{owner}/{repo}', u'emojis_url': u'https://api.github.com/emojis', u'hub_url': u'https://api.github.com/hub', u'keys_url': u'https://api.github.com/user/keys', u'following_url': u'https://api.github.com/user/following{/target}', u'emails_url': u'https://api.github.com/user/emails', u'authorizations_url': u'https://api.github.com/authorizations', u'code_search_url': u'https://api.github.com/search/code?q={query}{&page,per_page,sort,order}', u'followers_url': u'https://api.github.com/user/followers', u'public_gists_url': u'https://api.github.com/gists/public', u'organization_url': u'https://api.github.com/orgs/{org}', u'gists_url': u'https://api.github.com/gists{/gist_id}', u'feeds_url': u'https://api.github.com/feeds', u'user_search_url': u'https://api.github.com/search/users?q={query}{&page,per_page,sort,order}', u'user_url': u'https://api.github.com/users/{user}', u'events_url': u'https://api.github.com/events', u'organization_repositories_url': u'https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}', u'current_user_url': u'https://api.github.com/user', u'issue_search_url': u'https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}', u'notifications_url': u'https://api.github.com/notifications', u'starred_url': u'https://api.github.com/user/starred{/owner}{/repo}', u'starred_gists_url': u'https://api.github.com/gists/starred', u'current_user_authorizations_html_url': u'https://github.com/settings/connections/applications{/client_id}', u'user_repositories_url': u'https://api.github.com/users/{user}/repos{?type,page,per_page,sort}', u'team_url': u'https://api.github.com/teams'}

>>> response.json()['team_url']   #访问json的对象
u'https://api.github.com/teams'

九、下载图片/文件

图片/文件下载过程

浏览器模拟-->构建request(调试策略)-->读取流data-->打开文件，存入数据

有一些返回的context的内容是没有办法读取的，只有通过流的方式保存到文件中

import requests
def download_image():
    """
    demo:下载图片
    :return:
    """
    url= "http://img0.tech2ipo.com/upload/img/article/2015/06/1434420238601.png"
    headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'}
    response=requests.get(url,headers=headers,stream=True)
    with open('demo.png','wb') as fd:
        for chunk in response.iter_content(128):
            fd.write(chunk)
    print response.status_code
    # print  response.reason
    # print  response.content

　　

使用上下午文管理的方式，读物数据流，写入文件，最后关闭流

def download_image_improved():
    """
    demo:下载图片
    :return:
    """
    #伪造headers信息
    headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'}
    #限定url
    url= "http://img0.tech2ipo.com/upload/img/article/2015/06/1434420238601.png"
    response=requests.get(url,headers=headers,stream=True)
    from contextlib import closing
    with  closing(requests.get(url,headers=headers,stream=True)) as response:
        #打开文件
        with open('demo.png','wb') as fd:
            #每128写入一次
            for chunk in response.iter_content(128):
                fd.write(chunk)

　　

　

 十、事件钩子(Event hooks)

类似于js中的回调函数；

IO请求-->response-->回调函数

说白了，就是异步请求，request发出之后，直接返回，剩余的IO请求，由回调函数处理;

异步处理demo

import requests

def get_key_info(response,*args,**kwargs):
    """
    回调函数
    :return:
    """
    print response.headers['Content-Type']

def main():
    """
    主程序
    :return:
    """
    requests.get('https://api.github.com',hooks=dict(response=get_key_info))

main()

执行结果:
application/json; charset=utf-8

　　

 十一、HTTP认证

1、http基本认证

  requests a protected resource-->requests a username:password-->sends username:password-->returns requests resource

import requests

BASE_URL='https://api.github.com'

def construct_url(end_point):
    return '/'.join([BASE_URL,end_point])

def basic_auth():
    """
    基本认证
    :return:
    """

    response=requests.get(construct_url('user'),auth=('imoocdeo','imoocdeo123'))
    print response.text
    print response.request.headers

basic_auth()

打印结果:
{"message":"Bad credentials","documentation_url":"https://developer.github.com/v3"}
{'Authorization': 'Basic aW1vb2NkZW86aW1vb2NkZW8xMjM=', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0'}
这里字符串 aW1vb2NkZW86aW1vb2NkZW8xMjM=是使用base64编码的结果

在终端打开bpython(pip install bpython安装)

>>> import base64
>>> 
>>> base64.b64decode('aW1vb2NkZW86aW1vb2NkZW8xMjM=')
'imoocdeo:imoocdeo123'    #是用户名和密码拼凑起来的,但是这样可能不安全

　　

2、OAUTH认证

https://www.codewars.com/ 使用的登录认证方式

 APP---github

github生成access_token:   https://github.com/settings/tokens-->Generate new token-->选择该token拥有的权限

def basic_oauth():
    headers={'Authorization':'token 51684c8452ff83556f1a6705d2d85d7f68cd5ca7'}
    #user/emails,通过请求头把token带过去
    response=requests.get(construct_url('user/emails'),headers=headers)
    print response.request.headers
    print response.text
    print response.status_code
basic_oauth()

打印结果:
{'Authorization': 'token 51684c8452ff83556f1a6705d2d85d7f68cd5ca7', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0'}
[{"email":"qq_c123@163.com","primary":false,"verified":false,"visibility":null},{"email":"charles_ali@qq.com","primary":true,"verified":true,"visibility":"public"},{"email":"qq_c1231@163.com","primary":false,"verified":false,"visibility":null}]
200

　

使用python自带的语法糖类实现上述内容,__call__可以将实例化的对象当做类进行实例化调用,即将实例化的对象在加()，就可以调用__call__函数,f()();

from requests.auth import AuthBase

class GithubAuth(AuthBase):

    def __init__(self,token):
        self.token=token

    def __call__(self, r):
        #request 加headers
        r.headers['Authorization']=' '.join(['token',self.token])
        print r
        return r

def oauth_advaced():
    auth=GithubAuth('51684c8452ff83556f1a6705d2d85d7f68cd5ca7')
    response=requests.get(construct_url('user/emails'),auth=auth)
    print response.text

oauth_advaced()

调用结果:
<PreparedRequest [GET]>
[{"email":"qq_c123@163.com","primary":false,"verified":false,"visibility":null},{"email":"charles_ali@qq.com","primary":true,"verified":true,"visibility":"public"},{"email":"qq_c1231@163.com","primary":false,"verified":false,"visibility":null}]

　　

十二、requests库的proxy代理

import requests
proxies={'http':'sock5://127.0.0.1:1080','https':'sock5://127.0.0.1:1080'}    #设置代理地址

url='https://www.facebook.com'

response=requests.get(url,proxies=proxies,timeout=10)

　　

十三、Session和cookie

cookie:

浏览器请求(无cookie)-->服务器-->HTTP响应(响应头有set cookie,设置有效期，权限范围等等)-->解析cookie保存本地-->HTTP请求(携带cookie)--->解析cookie识别信息--HTTP响应

缺点:cookie是在浏览器端的，每次请求，都要携带cookie，消耗带宽;cookie可以伪造,不安全;

session:

浏览器HTTP请求-->服务器存储session-->HTTP响应(set cookie-session-id  叫做session id的cookie)-->解析cookie保存本地(这个cookie非常小,仅含有session id)-->解析session id(到存储中查，该session id对于的用户信息，权限信息是什么)-->HTTP响应