在更新了requests包之后,发现我电脑上的charles工具无法再成功抓取到数据包。百度了半年都没有找到原因。
然后 我使用了 google 查到了 charles的最新的文档发现。需要设置代理,不然流量过去居然无法被默认抓取系统所有流量的charles抓到。真是 神奇。
具体在 stackoverflow上看到 先上一个 原文地址。http://stackoverflow.com/questions/8287628/proxies-with-python-requests-module
然后 把内容转载过来 我尝试了第二种方法 在环境变量 export里面增加 代理地址。但是似乎没有什么用 但是 在代码中写代理的办法是好使的。具体:
The proxies
' dict syntax is {"protocol":"ip:port", ...}
. With it you can specify different (or the same) proxie(s) for requests using http, https, and ftp protocols:
http_proxy = "http://10.10.1.10:3128"
https_proxy = "https://10.10.1.11:1080"
ftp_proxy = "ftp://10.10.1.10:3128"
proxyDict = {
"http" : http_proxy,
"https" : https_proxy,
"ftp" : ftp_proxy
}
r = requests.get(url, headers=headers, proxies=proxyDict)
Deduced from the requests
documentation:
Parameters:
method
– method for the new Request object.url
– URL for the new Request object.
...proxies
– (optional) Dictionary mapping protocol to the URL of the proxy.
...
On linux you can also do this via the HTTP_PROXY
, HTTPS_PROXY
, and FTP_PROXY
environment variables:
export HTTP_PROXY=10.10.1.10:3128
export HTTPS_PROXY=10.10.1.11:1080
export FTP_PROXY=10.10.1.10:3128
On Windows:
set http_proxy=10.10.1.10:3128
set https_proxy=10.10.1.11:1080
set ftp_proxy=10.10.1.10:3128
Thanks, Jay for pointing this out:
The syntax changed with requests 2.0.0.
You'll need to add a schema to the url: http://docs.python-requests.org/en/latest/user/advanced/#proxies