day55——requests模块

requests模块是一个第三方模块，对urllib模块进行了封装，提供了更友好更强大的API

 

使用pip安装即可

 

requests模块的官方文档：http://docs.python-requests.org/zh_CN/latest/

 

1. GET请求

In [1]: import requests

In [2]: help(requests.get)                              //使用requests.get提交GET请求
Help on function get in module requests.api:

get(url, params=None, **kwargs)
    Sends a GET request.

    :param url: URL for the new :class:`Request` object.
    :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
    :param **kwargs: Optional arguments that ``request`` takes.
    :return: :class:`Response <Response>` object
    :rtype: requests.Response


In [3]: r = requests.get('https://github.com/timeline.json')


In [6]: dic = {'hostname':'n3', 'ip':'3.3.3.3'}                       

In [7]: r = requests.get('http://192.168.123.105:8000/db/', params=dic)           //GET请求携带参数，不需要进行编码

In [10]: r.text
Out[10]: u'GET OK'

In [11]: r.status_code
Out[11]: 200

In [12]: r.url
Out[12]: u'http://192.168.123.105:8000/db/?ip=3.3.3.3&hostname=n3'


In [11]: ua = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0'}

In [12]: r = requests.get('https://www.qiushibaike.com/', headers=ua)        //支持自定义User-Agent

In [13]: r.status_code
Out[13]: 200

2. POST请求

In [1]: import requests                           

In [2]: import json                                 

In [3]: url = 'http://httpbin.org/post'           

In [4]: ua = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0'}

In [5]: dic = {'key1': 'value1', 'key2': 'value2'}

In [6]: r = requests.post(url, data=dic)            //POST请求携带参数，不需要编码

In [7]: r.status_code
Out[7]: 200

In [8]: json.loads(r.text)
Out[8]:
{u'args': {},
 u'data': u'',
 u'files': {},
 u'form': {u'key1': u'value1', u'key2': u'value2'},
 u'headers': {u'Accept': u'*/*',
  u'Accept-Encoding': u'gzip, deflate',
  u'Connection': u'close',
  u'Content-Length': u'23',
  u'Content-Type': u'application/x-www-form-urlencoded',
  u'Host': u'httpbin.org',
  u'User-Agent': u'python-requests/2.18.4'},
 u'json': None,
 u'origin': u'119.129.228.119',
 u'url': u'http://httpbin.org/post'}

In [9]: r = requests.post(url, data=dic, headers=ua)            //同样支持自定义User-Agent

In [10]: r.status_code                               
Out[10]: 200

3. 响应内容

Requests 会自动解码来自服务器的内容。大多数 unicode 字符集都能被无缝地解码

当你收到一个响应时，Requests 会猜测响应的编码方式，用于在你调用 Response.text 方法时对响应进行解码。Requests 首先在 HTTP 头部检测是否存在指定的编码方式，如果不存在，则会使用 charade 来尝试猜测编码方式

你可以找出 Requests 使用了什么编码，并且能够使用 r.encoding 属性来改变它

 

3.1 文本型相应内容（取文本）

In [14]: r = requests.get('https://github.com/timeline.json')

In [15]: r.encoding
Out[15]: 'utf-8'

In [16]: r.text
Out[16]: u'{"message":"Hello there, wayfaring stranger. If youu2019re reading this then you probably didnu2019t see our blog post a couple of years back announcing that this API would go away: http://git.io/17AROg Fear not, you should be able to get what you need from the shiny new Events API instead.","documentation_url":"https://developer.github.com/v3/activity/events/#list-public-events"}'

3.2 二进制型响应内容（取图片，文件）

In [17]: r.content
Out[17]: '{"message":"Hello there, wayfaring stranger. If youxe2x80x99re reading this then you probably didnxe2x80x99t see our blog post a couple of years back announcing that this API would go away: http://git.io/17AROg Fear not, you should be able to get what you need from the shiny new Events API instead.","documentation_url":"https://developer.github.com/v3/activity/events/#list-public-events"}'

 3.3 原始套接字响应内容

In [21]: r = requests.get('https://github.com/timeline.json', stream=True)

In [22]: r.raw
Out[22]: <urllib3.response.HTTPResponse at 0x2d380d0>

In [23]: r.raw.read()
Out[23]: '{"message":"Hello there, wayfaring stranger. If youxe2x80x99re reading this then you probably didnxe2x80x99t see our blog post a couple of years back announcing that this API would go away: http://git.io/17AROg Fear not, you should be able to get what you need from the shiny new Events API instead.","documentation_url":"https://developer.github.com/v3/activity/events/#list-public-events"}'

3.4 当Requests不能自动解码时，自定义编码来解码

[root@web spider]# cat demon1.py
#!/usr/bin/env python

import requests

url = 'https://www.baidu.com/'
r = requests.get(url, verify=False)
print(r.encoding)                             //查看返回的文本的编码形式，在文本的头部有声明
print(r.text)


[root@web spider]# python demon1.py
ISO-8859-1
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css><title>ç¾åº¦ä¸ä¸ï¼ä½ å°±ç¥é</title>      //乱码

[root@web spider]# cat demon1.py   
#!/usr/bin/env python

import requests

url = 'https://www.baidu.com/'
r = requests.get(url, verify=False)
r.encoding = 'utf-8'                  //当requests模块不能正确对返回的内容解码时，需要定义r.encoding的值来解码，编码形式与返回的内容声明的编码形式一样
print(r.text)


[root@web spider]# python demon1.py
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title>

4. 响应状态码

In [33]: r = requests.get('http://httpbin.org/get')

In [34]: r.status_code
Out[34]: 200

In [35]: r.status_code == requests.codes.ok        //判断状态码是否为200
Out[35]: True

In [36]: 301 == requests.codes.ok                 
Out[36]: False

In [37]: 302 == requests.codes.ok                 
Out[37]: False

5. 响应头

In [48]: r = requests.get('https://www.github.com')             

In [49]: r.headers
Out[49]: {'Status': '200 OK', 'Expect-CT': 'max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"', 'X-Request-Id': 'fab96e29cdc0d636dd3ce5b51668717a', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'Content-Security-Policy': "default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src render.githubusercontent.com; connect-src 'self' uploads.github.com status.github.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src assets-cdn.github.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; img-src 'self' data: assets-cdn.github.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; media-src 'none'; script-src assets-cdn.github.com; style-src 'unsafe-inline' assets-cdn.github.com", 'X-Runtime-rack': '0.040115', 'Transfer-Encoding': 'chunked', 'Set-Cookie': 'logged_in=no; domain=.github.com; path=/; expires=Thu, 31 Dec 2037 04:34:58 -0000; secure; HttpOnly, _gh_sess=eyJzZXNzaW9uX2lkIjoiNzczMjViNjAxNzdkYmEyM2EwZjQ2NDkzMDBmZGM1ZmYiLCJsYXN0X3JlYWRfZnJvbV9yZXBsaWNhcyI6MTUxNDY5NDg5ODYyNywiX2NzcmZfdG9rZW4iOiJpZFVNTGlERHBDT0M2MHI4MXAyY2N4dmtYU21lTlFJUHVoSW9rTng4eC9nPSJ9--196e7163a50c84523f9bd24e5b1f411c39df0455; path=/; secure; HttpOnly', 'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload', 'Vary': 'X-PJAX, Accept-Encoding', 'Server': 'GitHub.com', 'X-GitHub-Request-Id': '8E25:166FF:3551BE5:4F1D6DD:5A4868F2', 'X-Runtime': '0.032924', 'X-UA-Compatible': 'IE=Edge,chrome=1', 'Cache-Control': 'no-cache', 'Date': 'Sun, 31 Dec 2017 04:34:58 GMT', 'X-Frame-Options': 'deny', 'Content-Type': 'text/html; charset=utf-8', 'Public-Key-Pins': 'max-age=0; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="RRM1dGqnDFsCJXBTHky16vi1obOlCgFFn/yOhI/y+ho="; pin-sha256="k2v657xBsOVe1PQRwOsHsw3bsGT2VzIqz5K+59sNQws="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="IQBnNBEiFuhj+8x6X8XLgh01V9Ic5/V3IRQLNFFc7v4="; pin-sha256="iie1VXtL7HzAMF+/PVPR9xzT80kQxdZeJ+zduCB3uj0="; pin-sha256="LvRiGEjRqfzurezaWuj8Wie2gyHMrW5Q06LspMnox7A="; includeSubDomains', 'Content-Encoding': 'gzip'}

In [51]: r.headers['Set-Cookie']
Out[51]: 'logged_in=no; domain=.github.com; path=/; expires=Thu, 31 Dec 2037 04:34:58 -0000; secure; HttpOnly, _gh_sess=eyJzZXNzaW9uX2lkIjoiNzczMjViNjAxNzdkYmEyM2EwZjQ2NDkzMDBmZGM1ZmYiLCJsYXN0X3JlYWRfZnJvbV9yZXBsaWNhcyI6MTUxNDY5NDg5ODYyNywiX2NzcmZfdG9rZW4iOiJpZFVNTGlERHBDT0M2MHI4MXAyY2N4dmtYU21lTlFJUHVoSW9rTng4eC9nPSJ9--196e7163a50c84523f9bd24e5b1f411c39df0455; path=/; secure; HttpOnly'

6. Cookie

cookie虽然不是字典类型，但是可以通过字典的方法进行取值 e.g. print(r.cookies['logged_in'])

cookie的五要素：

• name：对应的key值
• value：key对应的value
• domain：cookie所在的域，默认情况下时请求的域名
• path：cookie中的path能在域中进一步控制cookie的访问，当path=/，当前域的所有请求都可以访问这个cookie；如果path设为其他值，例如path=/test，那么只有www.domain.com/test下的请求才能访问这个cookie
• expires：cookie的过期时间，cookie作用于客户端（浏览器也是一个客户端），而session是作用于服务端，服务端的session为客户端生成一个cookie，session用于校验客户端携带的cookie访问的合法性

[root@web spider]# cat cookie1.py   
#!/usr/bin/env python

import requests

s = requests.session()
s.get('https://www.github.com')
print(s.cookies)
print('#'*30)
print(s.cookies.keys())
print('#'*30)
print(s.cookies.values())
print('#'*30)
for i in s.cookies:
    print(i.name, i.value, i.domain, i.path, i.expires)


[root@web spider]# python cookie1.py
<<class 'requests.cookies.RequestsCookieJar'>[<Cookie logged_in=no for .github.com/>, <Cookie _gh_sess=eyJzZXNzaW9uX2lkIjoiMTQ5ZTFhODdkN2MzYzNjYzZmODQ2MDdjMThlMzUwMjgiLCJsYXN0X3JlYWRfZnJvbV9yZXBsaWNhcyI6MTUxNjUxMDAzNDk4NiwiX2NzcmZfdG9rZW4iOiJtVmhsSDFJaURxUlRNN3dXSGxCZG5CU3RaRW5YZ2JNbkFlenRzeXFxY3pzPSJ9--b739efdc41973b7a3a976d5a8a0d9b7691ca68ab for github.com/>]>
##############################
['logged_in', '_gh_sess']
##############################
['no','eyJzZXNzaW9uX2lkIjoiMTQ5ZTFhODdkN2MzYzNjYzZmODQ2MDdjMThlMzUwMjgiLCJsYXN0X3JlYWRfZnJvbV9yZXBsaWNhcyI6MTUxNjUxMDAzNDk4NiwiX2NzcmZfdG9rZW4iOiJtVmhsSDFJaURxUlRNN3dXSGxCZG5CU3RaRW5YZ2JNbkFlenRzeXFxY3pzPSJ9--b739efdc41973b7a3a976d5a8a0d9b7691ca68ab']
##############################
('logged_in', 'no', '.github.com', '/', 2147662035)
('_gh_sess','eyJzZXNzaW9uX2lkIjoiMTQ5ZTFhODdkN2MzYzNjYzZmODQ2MDdjMThlMzUwMjgiLCJsYXN0X3JlYWRfZnJvbV9yZXBsaWNhcyI6MTUxNjUxMDAzNDk4NiwiX2NzcmZfdG9rZW4iOiJtVmhsSDFJaURxUlRNN3dXSGxCZG5CU3RaRW5YZ2JNbkFlenRzeXFxY3pzPSJ9--b739efdc41973b7a3a976d5a8a0d9b7691ca68ab', 'github.com', '/', None)

6.1 使用已知的cookie访问网站

[root@web spider]# cat cookie2.py
#!/usr/bin/env python

import requests

url = 'http://httpbin.org/cookies'
r = requests.get(url, cookies={'key1': 'value1', 'key2': 'value2'})         //get()提供了一个专门接收cookie的参数，参数接收一个字典
print(r.text)


[root@web spider]# python cookie2.py
{
  "cookies": {
    "key1": "value1",
    "key2": "value2"
  }
}

7. 会话对象

会话对象让你能够跨请求保持某些参数。它也会在同一个 Session 实例发出的所有请求之间保持 cookie

In [110]: s = requests.session()                        //创建一个会话对象          

In [111]: help(s.get)                                  //get和post方法用法同requests.get/requests.post
Help on method get in module requests.sessions:

get(self, url, **kwargs) method of requests.sessions.Session instance
    Sends a GET request. Returns :class:`Response` object.

    :param url: URL for the new :class:`Request` object.
    :param **kwargs: Optional arguments that ``request`` takes.
    :rtype: requests.Response


In [112]: help(s.post)
Help on method post in module requests.sessions:

post(self, url, data=None, json=None, **kwargs) method of requests.sessions.Session instance
    Sends a POST request. Returns :class:`Response` object.

    :param url: URL for the new :class:`Request` object.
    :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
    :param json: (optional) json to send in the body of the :class:`Request`.
    :param **kwargs: Optional arguments that ``request`` takes.
    :rtype: requests.Response


In [113]: r = s.get('http://192.168.123.105:8000/admin')

In [114]: r.status_code
Out[114]: 200

In [115]: r.cookies
Out[115]: <<class 'requests.cookies.RequestsCookieJar'>[Cookie(version=0, name='csrftoken', value='nDCM6HJnfOI10QazYD78vdEO2Gt2r6NO', port=None, port_specified=False, domain='192.168.123.105', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=1546137248, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False)]>

8. 使用代理

[root@web spider]# cat daili1.py
#!/usr/bin/env python

import requests

url = 'http://2017.ip138.com/ic.asp'
r = requests.get(url)
r.encoding = 'gb2312'
print(r.text)

proxies = {
    # 'https': 'http://122.72.18.35:80',
    'http': 'http://125.88.177.128:3128'
}
r1 = requests.get(url, proxies=proxies)
r1.encoding = 'gb2312'
print(r1.text)


[root@web spider]# python daili1.py
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=gb2312">
<title> 您的IP地址 </title>
</head>
<body style="margin:0px"><center>您的IP是：[113.68.17.83] 来自：广东省广州市 电信</center></body></html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=gb2312">
<title> 您的IP地址 </title>
</head>
<body style="margin:0px"><center>您的IP是：[119.129.229.185] 来自：广东省广州市 电信</center></body></html>