03、request 模块高级

requests高级部分

• 代理
• cookie
• 验证码的识别
• 模拟登陆

代理

• 代理概念：代理服务器。
• 作用：接受请求==》请求转发。
• 代理和爬虫之间的关联：
  • 可以使用请求转发的机制使得目的服务器接收到的请求对应ip的一个改变。
• 为什么要使用代理改变请求的ip地址？
  • 爬虫程序在短时间内对指定的服务器发起了一个高频的请求，则请求对应的ip可能会被目的服务器禁止。
• 代理的一些基础知识：
  • 代理的匿名度：
    • 透明代理：目的服务器知道你使用了代理机制并且也知道你的真实IP
    • 匿名代理：知道使用了代理机制，但是不知道你的真实ip
    • 高匿代理：不知道使用了代理，也不知道你的真实ip
  • 代理的类型：
    • http
    • https
• 免费代理ip：
  • 快代理
  • 西祠代理
  • www.goubanjia.com
  • 代理精灵
• 需求：测试一下代理是否会生效
  • 准备：
    • 代理ip和端口号
    • 测试的过程
• get/post方法中使用proxies请求代理的设置
  • proxies={'http':'ip:port'}

import requests
url = 'https://www.baidu.com/s?ie=utf-8&wd=ip'
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36'
}
page_text = requests.get(url=url,headers=headers,proxies={'https':'112.195.96.115:32092'}).text
with open('./ip.html','w',encoding='utf-8') as fp:
    fp.write(page_text)

• 对西祠代理发起一个高频的请求，使得本机ip被禁止，然后采用代理池解决代理的反爬
  • url:https://www.xicidaili.com/nn/1
• 代理池的构建：就是一个列表，列表中存储的是字典。每一个字典存储的键值{'http':'ip:port'}

all_ips = []
url = 'http://t.11jsq.com/index.php/api/entry?method=proxyServer.generate_api_url&packid=1&fa=0&fetch_key=&groupid=0&qty=30&time=1&pro=&city=&port=1&format=txt&ss=3&css=&dt=1&specialTxt=3&specialJson=&usertype=2'
page_text = requests.get(url,headers=headers).text
tree = etree.HTML(page_text)
ip_list = tree.xpath('//body//text()')[0].split('
')
for ip in ip_list:
    dic = {
        'https':ip
    }
    all_ips.append(dic)

from lxml import etree
import random
url = 'https://www.xicidaili.com/nn/%d'
ips = []
for page in range(1,3):
    new_url = format(url%page)
    page_text = requests.get(new_url,headers=headers,proxies=random.choice(all_ips)).text
    tree = etree.HTML(page_text)
    #在xpath表达式中一定不可以出现tbody标签
    tr_list = tree.xpath('//*[@id="ip_list"]//tr')[1:]
    for tr in tr_list:
        ip = tr.xpath('./td[2]/text()')[0]
        ips.append(ip)
print(len(ips))

200

cookie处理

• https://xueqiu.com/，对雪球网中的新闻数据进行爬取
• 分析：
  • 新闻数据是通过ajax动态加载出来的
  • 捕获到ajax数据包中的url
• 请求头中重要的头信息
  • User-Agent
  • Cookie
  • Referer
• cookie的处理
  • 手动处理
    • 将cookie作用到headers中即可
  • 自动处理
    • session = requests.Session()
    • session的作用：
      • session可以像requests模块一样调用get和post进行请求发送
      • 在进行请求发送的过程中如果产生了cookie，则cookie会被自动存储到session对象中。

session = requests.Session()
#第一次请求发送：为了捕获cookie且存储到session对象中
first_url = 'https://xueqiu.com/'
session.get(first_url,headers=headers)

#第二次请求发送：携带者cookie进行的请求发送
url = 'https://xueqiu.com/v4/statuses/public_timeline_by_category.json?since_id=-1&max_id=20358211&count=15&category=-1'
json_data = session.get(url=url,headers=headers).json()
json_data

{'addition': [],
 'list': [{'category': 0,
   'column': '沪深',
   'data': '{"id":136759791,"title":"与高瓴一起持三年格力不减仓","description":"刚查了一下，今年六月我清仓格力。那时候的原因是市盈近15倍，贵了。 十月二十九日，我大调仓，又接回了格力。当下全仓。30%的平安，20%的招商，15%的洋河，10%格力、10%的美的、5%的五粮液。接回格力的原因，也在六月暂时清空格力的时候说过，十二倍的格力我觉得便宜了。 接回来后，运气，马上逢...","target":"/8892483450/136759791","reply_count":30,"retweet_count":5,"topic_title":"与高瓴一起持三年格力不减仓","topic_desc":"刚查了一下，今年六月我清仓格力。那时候的原因是市盈近15...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":null,"pic_size":{"width":2,"height":1},"pic":"","pic_sizes":[],"cover_pic":null,"user":{"id":8892483450,"profile":"/8892483450","description":"一、投资，说穿了只干一件事，那就是估值。\n    二、投资的原则说穿了只有两点，那就是低买，高卖。\n    三、投资无非投三好，那就是好行业、好公司、好价格。  \n    四、做好投资无非踏踏实实地做好四条：1，阅读；2研究；3表达。4.交流。\n     这就构成了我的价值分析1234：就是估值；低买、高卖；好行业、好公司、好价格；阅读、研究、表达、交流。","following":false,"screen_name":"价值分析1234","profile_image_url":"community/default/avatar.png,community/default/avatar.png!180x180.png,community/default/avatar.png!50x50.png,community/default/avatar.png!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[],"followers_count":824},"promotion":false,"answers":null,"view_count":336048,"created_at":1575381782000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"Android","retweeted_status":null,"like_count":26,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":8892483450,"longTextForIOS":false}',
   'id': 20358210,
   'original_status': None},
  {'category': 0,
   'column': '基金',
   'data': '{"id":136752039,"title":"银行股指估值接近历史最低：抄底银行ETF机构资金大揭秘！","description":"2019年还有不到一个月结束，你完成今年的投资目标了嘛？回顾今年以来的行情，年内上涨表现亮眼的家用电器、食品饮料、医药生物等板块背后均是资金的抱团。与过去不同，目前A股投资逻辑生变，长线资金越来越成为投资风向标。 值得注意的是，经过前期的上涨，家用电器、食品饮料等板块估值较高，业...","target":"/1815461130/136752039","reply_count":12,"retweet_count":8,"topic_title":"银行股指估值接近历史最低：抄底银行ETF机构资金大揭秘！","topic_desc":"2019年还有不到一个月结束，你完成今年的投资目标了嘛？回...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":"https://xqimg.imedao.com/16ecb76eb5ebd8e3fa909370.png!186x186.jpg","pic_size":{"width":2,"height":1},"pic":"https://xqimg.imedao.com/16ecb76eb5ebd8e3fa909370.png!345x210.webp,https://xqimg.imedao.com/16ecb770d66bd943febffeca.png!345x210.webp,https://xqimg.imedao.com/16ecb773218bfe63fe5b58ac.png!345x210.webp","pic_sizes":[{"width":145,"height":524},{"width":554,"height":217},{"width":554,"height":416}],"cover_pic":null,"user":{"id":1815461130,"profile":"/1815461130","description":"","following":false,"screen_name":"究极马里奥","profile_image_url":"community/20188/1537324472738-1537324476843.jpg,community/20188/1537324472738-1537324476843.jpg!180x180.png,community/20188/1537324472738-1537324476843.jpg!50x50.png,community/20188/1537324472738-1537324476843.jpg!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[],"followers_count":1756},"promotion":false,"answers":null,"view_count":325182,"created_at":1575371901000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"雪球","retweeted_status":null,"like_count":11,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":1815461130,"longTextForIOS":false}',
   'id': 20358209,
   'original_status': None},
  {'category': 0,
   'column': '沪深',
   'data': '{"id":136602580,"title":"2019年1-11月中国房地产企业销售TOP100排行榜","description":"导 读 1、年底加大供应，TOP100房企整体环比增11% ； 2、TOP30房企中，20家环比业绩上升； 3、TOP10权益销售门槛增幅最大达15%； 4、9家房企提前完成目标，部分房企仍有压力。 ☉文／克而瑞研究中心 榜单改版说明 今年是克而瑞编制中国房地产企业销售排行榜的第11个年头，从最初的TOP20到TOP50、T...","target":"/1934804717/136602580","reply_count":41,"retweet_count":30,"topic_title":"2019年1-11月中国房地产企业销售TOP100排行榜","topic_desc":"导 读 1、年底加大供应，TOP100房企整体环比增11% ； 2、TO...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":"https://xqimg.imedao.com/16ebbe26a6856653feb0acd4.jpeg!186x186.jpg","pic_size":{"width":2,"height":1},"pic":"https://xqimg.imedao.com/16ebbe26a6856653feb0acd4.jpeg!345x210.webp,https://xqimg.imedao.com/16ebbe26ddc57763fe9f92b6.jpeg!345x210.webp,https://xqimg.imedao.com/16ebbe2716f571e3fe6e1842.png!345x210.webp,https://xqimg.imedao.com/16ebbe272e257773fe198635.png!345x210.webp,https://xqimg.imedao.com/16ebbe273e457783fc1957a6.png!345x210.webp,https://xqimg.imedao.com/16ebbe274fe56663feff0e7f.png!345x210.webp","pic_sizes":[{"width":659,"height":4432},{"width":659,"height":4490},{"width":573,"height":384},{"width":551,"height":370},{"width":558,"height":315},{"width":592,"height":371},{"width":71,"height":14}],"cover_pic":null,"user":{"id":1934804717,"profile":"/1934804717","description":"克而瑞研究中心作为易居企业集团专业研发部门，在中国房地产行业研究领域的排名历年来保持领先。成立12年来，每年成果出品超百余份，规模以百万字计。聚焦房地产宏观、市场、土地、城市、企业战略、企业金融等十大研究方向。迄今为止，连续11年发布中国房地产企业销售TOP100排行榜，并不断扩容至TOP200，备受房企、基金、券商、政府相关部门认可。","following":false,"screen_name":"克而瑞地产研究","profile_image_url":"community/20182/1521088654031-1521088657129.jpg,community/20182/1521088654031-1521088657129.jpg!180x180.png,community/20182/1521088654031-1521088657129.jpg!50x50.png,community/20182/1521088654031-1521088657129.jpg!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[{"verified_type":"8","verified_desc":"地产行业自媒体克而瑞地产研究"}],"followers_count":19719},"promotion":false,"answers":null,"view_count":213973,"created_at":1575110211000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"雪球","retweeted_status":null,"like_count":25,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":1934804717,"longTextForIOS":false}',
   'id': 20358208,
   'original_status': None},
  {'category': 0,
   'column': '沪深',
   'data': '{"id":136760743,"title":"2019年11月总结以及展望2020年","description":"投资以来的年收益 截止2019年12月2日，今年收益率达到55.4%，10年以来投资以来总收益达到722%，年化21.9%。实盘组合如下 实盘组合 2号受<a href=\"http://xueqiu.com/S/01918\" target=\"_blank\">$融创中国(01918)$</a>大幅上涨影响，收益率有了较大幅度的增长。借用<a href=\"http://xueqiu.com/n/正合奇胜天舒\" target=\"_blank\">@正合奇胜天舒</a> 《中国恒大价值投资案列》里的观点融创基本面的良好和同时良好技术形态发生共...","target":"/6451611049/136760743","reply_count":47,"retweet_count":10,"topic_title":"2019年11月总结以及展望2020年","topic_desc":"投资以来的年收益 截止2019年12月2日，今年收益率达到55.4%...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":"https://xqimg.imedao.com/16ecc1539cac6293fdbc9079.jpeg!186x186.jpg","pic_size":{"width":2,"height":1},"pic":"https://xqimg.imedao.com/16ecc1539cac6293fdbc9079.jpeg!345x210.webp,https://xqimg.imedao.com/16ecc153cc1c62a3fe7836a8.jpeg!345x210.webp,https://xqimg.imedao.com/16ecc153b94c4813fe279ff4.jpeg!345x210.webp,https://xqimg.imedao.com/16ecc153c01c6ee3f8a245a6.jpeg!345x210.webp,https://xqimg.imedao.com/16ecc17d9a1c5c33fe0f03ac.jpeg!345x210.webp","pic_sizes":[{"width":1488,"height":1984},{"width":1488,"height":1984},{"width":1080,"height":2244},{"width":1080,"height":2244},{"width":1080,"height":2244}],"cover_pic":null,"user":{"id":6451611049,"profile":"/6451611049","description":"《投资者说》2019年1月访谈嘉宾，雪球保险股研究牛人，公众号“汾水之畔”，10年开始入市，10年投资美的，格力电器，12-15投资招商银行，兴业银行，浦发银行，15年开始介入港股，15-16投资港股交通银行，光大银行，16年投资中国太平，中国平安，17年投资融创中国，截止2019年4月，总收益率700%以上","following":false,"screen_name":"润哥","profile_image_url":"community/20192/1553691059638-1553691067746.png,community/20192/1553691059638-1553691067746.png!180x180.png,community/20192/1553691059638-1553691067746.png!50x50.png,community/20192/1553691059638-1553691067746.png!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[{"verified_type":"5","verified_desc":"用户已完成实名身份认证"}],"followers_count":13388},"promotion":false,"answers":null,"view_count":3380959,"created_at":1575382899000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"Android","retweeted_status":null,"like_count":56,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":6451611049,"longTextForIOS":false}',
   'id': 20358207,
   'original_status': None},
  {'category': 0,
   'column': '沪深',
   'data': '{"id":136754794,"title":"暴力扭亏！长安汽车2019年盈利预测","description":"我们之前在这篇文章中<a href=\"https://xueqiu.com/9930706260/136602635\" title=\"https://xueqiu.com/9930706260/136602635\" target=\"_blank\" class=\"status-link\">网页链接</a> 提到关于长安出售PSA后的全年盈利大致情况： 单纯只考虑出售PSA的情况下，2019年全年保守估计长安汽车亏损10.2亿或盈利67.2亿。 今天长安汽车又释放一大利好，新能源公司引入战略投资后增厚长安汽车利润22.91亿。 <a href=\"https://xueqiu.com/S/SZ000625/136752514?from=groupmessage&isappinstalled=0\" title=\"https://xueqiu.com/S/SZ000625/136752514?from=groupmessage&isappinstalled=0\" target=\"_blank\" class=\"status-link\">网页链接</a>","target":"/9930706260/136754794","reply_count":83,"retweet_count":2,"topic_title":"暴力扭亏！长安汽车2019年盈利预测","topic_desc":"我们之前在这篇文章中 网页链接 提到关于长安出售PSA后的全...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":"https://xqimg.imedao.com/16ecba39e3cc00f3fdf6c472.jpg!186x186.jpg","pic_size":{"width":2,"height":1},"pic":"https://xqimg.imedao.com/16ecba39e3cc00f3fdf6c472.jpg!345x210.webp,https://xqimg.imedao.com/16ecba1ae89c17a3fde286bc.jpg!345x210.webp","pic_sizes":[{"width":2048,"height":1286},{"width":2048,"height":1469}],"cover_pic":null,"user":{"id":9930706260,"profile":"/9930706260","description":"模型、周期、概率、置信区间、决策树、随机森林、第二层思维、嫉妒、自负、盲从......","following":false,"screen_name":"run_rabbit_run","profile_image_url":"community/20199/1572439227837-1572439228092.jpg,community/20199/1572439227837-1572439228092.jpg!180x180.png,community/20199/1572439227837-1572439228092.jpg!50x50.png,community/20199/1572439227837-1572439228092.jpg!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[{"verified_type":"5","verified_desc":"用户已完成实名身份认证"}],"followers_count":911},"promotion":false,"answers":null,"view_count":3445296,"created_at":1575375494000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"iPhone","retweeted_status":null,"like_count":19,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":9930706260,"longTextForIOS":false}',
   'id': 20358206,
   'original_status': None},
  {'category': 0,
   'column': '沪深',
   'data': '{"id":136752943,"title":"发生在身边的股市（14）：如何运用借贷开通证券所有操作权限","description":"我叫牧尘，我很慌… 三十来岁背负着几十万的借贷炒股，每天面临着随时爆仓的压力前行，我的精神压力巨大！！！ 我迫切想早点实现财富目标，但又怕这个“死亡游戏”随时会结束，踏在这条前途未卜的道上，人生第一次那么深刻的感受到什么是：生死有命，富贵由天！ <a href=\"https://xueqiu.com/8878684550/135769002\" title=\"https://xueqiu.com/8878684550/135769002\" target=\"_blank\">发生在身边的股市（13）-第一个百万目标很难吗？</a>","target":"/8878684550/136752943","reply_count":116,"retweet_count":4,"topic_title":"发生在身边的股市（14）：如何运用借贷开通证券所有操作权限","topic_desc":"我叫牧尘，我很慌… 三十来岁背负着几十万的借贷炒股，每天...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":"https://xqimg.imedao.com/16ecb7246eebe203fe7dec26.jpeg!186x186.jpg","pic_size":{"width":2,"height":1},"pic":"https://xqimg.imedao.com/16ecb7246eebe203fe7dec26.jpeg!345x210.webp","pic_sizes":[{"width":1080,"height":2340}],"cover_pic":null,"user":{"id":8878684550,"profile":"/8878684550","description":"我很慌，我正做着一件所有人一直想却不敢做的“死亡游戏”：冒险借贷炒股。踏在这条前途未卜的道上，生死有命，富贵由天。","following":false,"screen_name":"努力养家的猪","profile_image_url":"community/20199/1572411744879-1572411745090.jpeg,community/20199/1572411744879-1572411745090.jpeg!180x180.png,community/20199/1572411744879-1572411745090.jpeg!50x50.png,community/20199/1572411744879-1572411745090.jpeg!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[],"followers_count":1228},"promotion":false,"answers":null,"view_count":3429115,"created_at":1575373197000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"Android","retweeted_status":null,"like_count":51,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":8878684550,"longTextForIOS":false}',
   'id': 20358205,
   'original_status': None},
  {'category': 0,
   'column': '沪深',
   'data': '{"id":136752508,"title":"关于近期医药股调整的一些想法","description":"随着最新医保谈判落地，医药股进行了一轮深度调整，不但以仿制药为主的公司股价创新低，创新药大白马也大幅杀跌。一些之前信心满满的价值投机者也开始动摇了。 与很多人相反，在详细整理了一些数据后，我认为现在是一个好的上车机会。 首先梳理一下市场里下跌原因的分析： 1. 医保谈判价格远低于...","target":"/9827772605/136752508","reply_count":49,"retweet_count":22,"topic_title":"关于近期医药股调整的一些想法","topic_desc":"随着最新医保谈判落地，医药股进行了一轮深度调整，不但以...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":"https://xqimg.imedao.com/16ecb67e25dbe9a3fcadbc32.png!186x186.jpg","pic_size":{"width":2,"height":1},"pic":"https://xqimg.imedao.com/16ecb67e25dbe9a3fcadbc32.png!345x210.webp,https://xqimg.imedao.com/16ecb671bbabd923f96e051a.png!345x210.webp","pic_sizes":[{"width":923,"height":418},{"width":790,"height":552},{"width":1248,"height":183}],"cover_pic":null,"user":{"id":9827772605,"profile":"/9827772605","description":null,"following":false,"screen_name":"白马笑西风nby","profile_image_url":"community/201911/1575376753488-1575376753636.jpeg,community/201911/1575376753488-1575376753636.jpeg!180x180.png,community/201911/1575376753488-1575376753636.jpeg!50x50.png,community/201911/1575376753488-1575376753636.jpeg!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[],"followers_count":113},"promotion":false,"answers":null,"view_count":3414687,"created_at":1575372633000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"雪球","retweeted_status":null,"like_count":61,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":9827772605,"longTextForIOS":false}',
   'id': 20358204,
   'original_status': None},
  {'category': 0,
   'column': '沪深',
   'data': '{"id":136756819,"title":"正邦科技11月出栏数据个人解读","description":"<a href=\"http://xueqiu.com/S/SZ002157\" target=\"_blank\">$正邦科技(SZ002157)$</a><a href=\"http://xueqiu.com/S/SZ002124\" target=\"_blank\">$天邦股份(SZ002124)$</a><a href=\"http://xueqiu.com/S/SZ002714\" target=\"_blank\">$牧原股份(SZ002714)$</a>今天晚上出乎很多人的意料，正邦科技提前公布了11月的出栏月报。总体来看，我觉得还不错，至少符合甚至小幅超出了我的预期。 首先，要表扬公司的透明化程度大幅提高，从上个月的商品猪均价，到这个月的出栏均重，专业化水平不断提高...","target":"/3778711419/136756819","reply_count":38,"retweet_count":7,"topic_title":"正邦科技11月出栏数据个人解读","topic_desc":"<a href=\"http://xueqiu.com/S/SZ002157\"   target=\"_blank\">$正邦科技(SZ002157)$</a> <a href=\"http://xueqiu.com/S/SZ002124\"   target=\"_blank\">$天邦股份(SZ002124)$</a> $牧原股份(SZ0...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":null,"pic_size":{"width":2,"height":1},"pic":"","pic_sizes":[],"cover_pic":null,"user":{"id":3778711419,"profile":"/3778711419","description":"","following":false,"screen_name":"只为猪八","profile_image_url":"community/20198/1567899913411-1567899913684.jpg,community/20198/1567899913411-1567899913684.jpg!180x180.png,community/20198/1567899913411-1567899913684.jpg!50x50.png,community/20198/1567899913411-1567899913684.jpg!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[],"followers_count":876},"promotion":false,"answers":null,"view_count":3366020,"created_at":1575378018000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"雪球","retweeted_status":null,"like_count":48,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":3778711419,"longTextForIOS":false}',
   'id': 20358203,
   'original_status': None},
  {'category': 0,
   'column': '沪深',
   'data': '{"id":136757478,"title":"烽火/克来转债中签号公布，华锋转债周三申购","description":"昨夜美股大跌，今天A股低开后小幅震荡，午后反弹，涨0.31%收于2884.70点。沪市成交量继续萎靡，不足1400亿元。 市场短期走势，不可预测。涨涨跌跌，跌跌涨涨，上下波动的是你的心。继续来说可转债吧。 1、烽火转债和克来转债中签号公布 烽火转债，网上中签率为0.01999166%。打满近20%的概率中10张...","target":"/1910558321/136757478","reply_count":43,"retweet_count":1,"topic_title":"烽火/克来转债中签号公布，华锋转债周三申购","topic_desc":"昨夜美股大跌，今天A股低开后小幅震荡，午后反弹，涨0.31%...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":"https://xqimg.imedao.com/16ecbe444e3c34e3fca0fdd2.jpeg!186x186.jpg","pic_size":{"width":2,"height":1},"pic":"https://xqimg.imedao.com/16ecbe444e3c34e3fca0fdd2.jpeg!345x210.webp,https://xqimg.imedao.com/16ecbe487f7c2593fe860671.jpeg!345x210.webp,https://xqimg.imedao.com/16ecbe4c0cfc3ea3fec31fd9.jpg!345x210.webp,https://xqimg.imedao.com/16ecbe50593c3333fd6803e6.jpeg!345x210.webp,https://xqimg.imedao.com/16ecbe53a51c3ec3fcd43d9a.jpg!345x210.webp,https://xqimg.imedao.com/16ecbe5724bc4b33fcd52049.jpg!345x210.webp,https://xqimg.imedao.com/16ecbe5a99cc3ef3fe94d346.jpeg!345x210.webp,https://xqimg.imedao.com/16ecbf0a3efc47a3fe9ed011.png!345x210.webp","pic_sizes":[{"width":1080,"height":1921},{"width":1080,"height":1921},{"width":1368,"height":444},{"width":1080,"height":1921},{"width":870,"height":486},{"width":647,"height":582},{"width":1080,"height":1921},{"width":600,"height":30},{"width":1000,"height":1000}],"cover_pic":null,"user":{"id":1910558321,"profile":"/1910558321","description":"职业投资者，钟爱可转债。","following":false,"screen_name":"鲨鱼投资者","profile_image_url":"community/20194/1559197225425-1559197225819.jpg,community/20194/1559197225425-1559197225819.jpg!180x180.png,community/20194/1559197225425-1559197225819.jpg!50x50.png,community/20194/1559197225425-1559197225819.jpg!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[{"verified_type":"5","verified_desc":"用户已完成实名身份认证"}],"followers_count":13962},"promotion":false,"answers":null,"view_count":3358174,"created_at":1575378927000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"雪球","retweeted_status":null,"like_count":33,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":1910558321,"longTextForIOS":false}',
   'id': 20358202,
   'original_status': None},
  {'category': 0,
   'column': '沪深',
   'data': '{"id":136757071,"title":"大A股又挣扎了一下","description":"又到天桥底下说书时间。 今天大A股又顽强的抵抗了一把，本来昨晚外围算得上大跌的情况下，早上日经，恒生都低开不少，大A股也跟着虚脱了一下，没想到尾盘还硬着收红。在这个时候行情难熬，有子弹的人，就希望多跌点。没子弹的人，感觉这日子无聊极了，是一种煎熬。特别是对市场核心资产，估计不少...","target":"/8592131633/136757071","reply_count":40,"retweet_count":8,"topic_title":"大A股又挣扎了一下","topic_desc":"又到天桥底下说书时间。 今天大A股又顽强的抵抗了一把，本...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":"https://xqimg.imedao.com/16ecbde7e58c3963fde296e9.png!186x186.jpg","pic_size":{"width":2,"height":1},"pic":"https://xqimg.imedao.com/16ecbde7e58c3963fde296e9.png!345x210.webp","pic_sizes":[{"width":1080,"height":514},{"width":734,"height":179}],"cover_pic":null,"user":{"id":8592131633,"profile":"/8592131633","description":"","following":false,"screen_name":"孥孥的大树","profile_image_url":"community/20196/1563788737495-1563788739672.jpg,community/20196/1563788737495-1563788739672.jpg!180x180.png,community/20196/1563788737495-1563788739672.jpg!50x50.png,community/20196/1563788737495-1563788739672.jpg!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[{"verified_type":"3","verified_desc":"房地产行业研究达人"},{"verified_type":"1","verified_desc":"雪球2019-2018年度十大内容贡献奖得主"},{"verified_type":"5","verified_desc":"用户已完成实名身份认证"}],"followers_count":116157},"promotion":false,"answers":null,"view_count":3391472,"created_at":1575378395000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"雪球","retweeted_status":null,"like_count":99,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":8592131633,"longTextForIOS":false}',
   'id': 20358201,
   'original_status': None},
  {'category': 0,
   'column': '沪深',
   'data': '{"id":136735141,"title":"股价=市盈率*每股利润","description":"这篇文章介绍股票最基础也是最重要的公式。以下数据都是大概，因为手机写文章不追求精确（文章是之前写的，数据比较老）。 股价=市盈率*每股收益 比如工行的股价5.3 其市盈率5.97 每股收益三个季度0.67，动态计算全年为0.89 股价5.3=市盈率5.97*每股收益0.89 有人说，这个公式不就是市盈率的定义...","target":"/8808125263/136735141","reply_count":10,"retweet_count":13,"topic_title":"股价=市盈率*每股利润","topic_desc":"这篇文章介绍股票最基础也是最重要的公式。以下数据都是大...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":null,"pic_size":{"width":2,"height":1},"pic":"","pic_sizes":[],"cover_pic":null,"user":{"id":8808125263,"profile":"/8808125263","description":"雪球，知乎，公众号，我都叫洛阳小散户。","following":false,"screen_name":"洛阳小散户","profile_image_url":"community/20181/1518920564408-1518920564691.jpeg,community/20181/1518920564408-1518920564691.jpeg!180x180.png,community/20181/1518920564408-1518920564691.jpeg!50x50.png,community/20181/1518920564408-1518920564691.jpeg!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[{"verified_type":"6","verified_desc":"雪球沪深实盘认证用户"},{"verified_type":"5","verified_desc":"用户已完成实名身份认证"}],"followers_count":14280},"promotion":false,"answers":null,"view_count":3656350,"created_at":1575357183000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"雪球","retweeted_status":null,"like_count":49,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":8808125263,"longTextForIOS":false}',
   'id': 20358200,
   'original_status': None},
  {'category': 0,
   'column': '沪深',
   'data': '{"id":136738859,"title":"格力电器416亿股权改革，终于靴子落地混改后几大变化？","description":"<a href=\"http://xueqiu.com/n/今日话题\" target=\"_blank\">@今日话题</a> <a href=\"http://xueqiu.com/n/雪球达人秀\" target=\"_blank\">@雪球达人秀</a> 格力电器12月2日晚公告，公司控股股东格力集团与珠海明骏签署《股份转让协议》，约定珠海明骏以46.17元/股的价格受让格力集团持有的格力电器9.02亿股股份（占格力电器总股本的15%），合计转让价款为416.62亿元。备受关注的管理层股权激励方面，公告显示，珠海高瓴等同意，...","target":"/1566609429/136738859","reply_count":11,"retweet_count":3,"topic_title":"格力电器416亿股权改革，终于靴子落地混改后几大变化？","topic_desc":"<a href=\"http://xueqiu.com/n/今日话题\"  target=\"_blank\">@今日话题</a> <a href=\"http://xueqiu.com/n/雪球达人秀\"  target=\"_blank\">@雪球达人秀</a> 格力电器12月2日晚公告，公司控股股...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":null,"pic_size":{"width":2,"height":1},"pic":"","pic_sizes":[],"cover_pic":null,"user":{"id":1566609429,"profile":"/1566609429","description":"不以涨喜不以跌悲，低于价值买高于价值卖不管牛市还是熊市都一样(逆向思维)\n天若有情天亦老 人间正道是沧桑","following":false,"screen_name":"佛系小资","profile_image_url":"community/201811/1545889172613-1545889172896.jpeg,community/201811/1545889172613-1545889172896.jpeg!180x180.png,community/201811/1545889172613-1545889172896.jpeg!50x50.png,community/201811/1545889172613-1545889172896.jpeg!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[],"followers_count":4851},"promotion":false,"answers":null,"view_count":3619271,"created_at":1575360231000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"雪球","retweeted_status":null,"like_count":21,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":1566609429,"longTextForIOS":false}',
   'id': 20358199,
   'original_status': None},
  {'category': 0,
   'column': '沪深',
   'data': '{"id":136696900,"title":"核心资产暴跌，该买还是该卖？","description":"前言：最近两三年，A股市场分化很大，大部分个股是一路震荡下跌。另一小部分，被称为“龙头溢价”，后进化改为“核心资产”的优质公司，则维持震荡持续上涨。 可以用一句话总结，这段时期是大部分股的熊市，同时是小部分股的牛市。这些小部分公司在这两三年涨幅高达几倍，是市场最耀眼的明星。 然...","target":"/3241529995/136696900","reply_count":11,"retweet_count":9,"topic_title":"核心资产暴跌，该买还是该卖？","topic_desc":"前言：最近两三年，A股市场分化很大，大部分个股是一路震荡...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":"https://xqimg.imedao.com/16ec720095b9ab93fc9bf367.jpg!186x186.jpg","pic_size":{"width":2,"height":1},"pic":"https://xqimg.imedao.com/16ec720095b9ab93fc9bf367.jpg!345x210.webp,https://xqimg.imedao.com/16ec72009909aba3feb499ec.jpg!345x210.webp","pic_sizes":[{"width":804,"height":462},{"width":760,"height":539}],"cover_pic":null,"user":{"id":3241529995,"profile":"/tiandaoqiniu","description":"公众号：tdqn666","following":false,"screen_name":"天道骑牛","profile_image_url":"community/20198/1568863596283-1568863602305.jpg,community/20198/1568863596283-1568863602305.jpg!180x180.png,community/20198/1568863596283-1568863602305.jpg!50x50.png,community/20198/1568863596283-1568863602305.jpg!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[{"verified_type":"3","verified_desc":"成长股投资研究达人"},{"verified_type":"1","verified_desc":"雪球星计划热门作者奖得主  《穿越牛熊：一个散户的股市投资修为路》作者 何银"},{"verified_type":"5","verified_desc":"用户已完成实名身份认证"}],"followers_count":146528},"promotion":false,"answers":null,"view_count":3611935,"created_at":1575298905000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"雪球","retweeted_status":null,"like_count":59,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":3241529995,"longTextForIOS":false}',
   'id': 20358198,
   'original_status': None},
  {'category': 0,
   'column': '沪深',
   'data': '{"id":136745411,"title":"公司比较系列：赣锋锂业和天齐锂业怎么选","description":"<a href=\"http://xueqiu.com/S/SZ002460\" target=\"_blank\">$赣锋锂业(SZ002460)$</a> <a href=\"http://xueqiu.com/S/SZ002466\" target=\"_blank\">$天齐锂业(SZ002466)$</a> 一、锂钴板块弹性最大 如果按照上中下游的弹性去比较的话，上游弹性是最大的（图片各板块净利率来源于长江证券）；2015-2017年锂和钴是新能源汽车板块最受益的； 很多人可能会问，历史是这样，并不代表未来仍然是这样；确实是这样，未来并不会简单重复...","target":"/8031840682/136745411","reply_count":40,"retweet_count":12,"topic_title":"公司比较系列：赣锋锂业和天齐锂业怎么选","topic_desc":"<a href=\"http://xueqiu.com/S/SZ002460\"   target=\"_blank\">$赣锋锂业(SZ002460)$</a> <a href=\"http://xueqiu.com/S/SZ002466\"   target=\"_blank\">$天齐锂业(SZ002466)$</a> 一、锂钴板块...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":"https://xqimg.imedao.com/16ecaeaf2e3b5be3fe437b65.png!186x186.jpg","pic_size":{"width":2,"height":1},"pic":"https://xqimg.imedao.com/16ecaeaf2e3b5be3fe437b65.png!345x210.webp,https://xqimg.imedao.com/16ecaf3e415b6a63fe2a3b8e.png!345x210.webp,https://xqimg.imedao.com/16ecaffd0a4b7363fc892eb4.png!345x210.webp","pic_sizes":[{"width":449,"height":223},{"width":752,"height":467},{"width":880,"height":318},{"width":892,"height":160},{"width":975,"height":183}],"cover_pic":null,"user":{"id":8031840682,"profile":"/8031840682","description":"行业研究员 实盘投资","following":false,"screen_name":"New-Energy研究","profile_image_url":"community/201811/1544087115052-1544087128961.jpg,community/201811/1544087115052-1544087128961.jpg!180x180.png,community/201811/1544087115052-1544087128961.jpg!50x50.png,community/201811/1544087115052-1544087128961.jpg!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[],"followers_count":827},"promotion":false,"answers":null,"view_count":487569,"created_at":1575365254000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"雪球","retweeted_status":null,"like_count":19,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":8031840682,"longTextForIOS":false}',
   'id': 20358197,
   'original_status': None},
  {'category': 0,
   'column': '沪深',
   'data': '{"id":136755541,"title":"中华历史五千年，最近A股波动二百点","description":"这几个月，A股真是在一个很小的空间里面不断震荡，上下200点并非戏言。那么如果继续下去，投资者怎样赚钱那？ 正文： 以上证为例，上证指数这几个月基本上龟缩在一个200点左右的空间里面震荡，年初一轮2400多点开始的行情涨到3200多点，然后就调整，调整以来就反反复复，就2800-3000这个区间，跌...","target":"/1062883669/136755541","reply_count":23,"retweet_count":5,"topic_title":"中华历史五千年，最近A股波动二百点","topic_desc":"这几个月，A股真是在一个很小的空间里面不断震荡，上下200...","topic_symbol":null,"topic_pic":null,"topic_pic_hd":null,"pic_type":0,"first_pic":"https://xqimg.imedao.com/16ecbbb7b01c11f3fee80560.jpg!186x186.jpg","pic_size":{"width":2,"height":1},"pic":"https://xqimg.imedao.com/16ecbbb7b01c11f3fee80560.jpg!345x210.webp,https://xqimg.imedao.com/16ecbbbdfa4c2103fe61bb32.jpg!345x210.webp,https://xqimg.imedao.com/16ecbbc266bc1be3fe489d4c.jpg!345x210.webp,https://xqimg.imedao.com/16ecbbc96dfc2233fc1e2a5b.jpeg!345x210.webp,https://xqimg.imedao.com/16ecbbcd515c1353fcd98e69.jpg!345x210.webp","pic_sizes":[{"width":836,"height":762},{"width":819,"height":663},{"width":653,"height":334},{"width":863,"height":1300},{"width":836,"height":700}],"cover_pic":null,"user":{"id":1062883669,"profile":"/1062883669","description":"你全家都价值投资","following":false,"screen_name":"青春的泥沼","profile_image_url":"community/20190/1548296765326-1548296786519.jpg,community/20190/1548296765326-1548296786519.jpg!180x180.png,community/20190/1548296765326-1548296786519.jpg!50x50.png,community/20190/1548296765326-1548296786519.jpg!30x30.png","photo_domain":"http://xavatar.imedao.com/","verified_infos":[{"verified_type":"3","verified_desc":"基金领域研究达人"},{"verified_type":"1","verified_desc":"雪球岛系列丛书《买基金小红书》作者"},{"verified_type":"6","verified_desc":"雪球沪深实盘认证用户"},{"verified_type":"5","verified_desc":"用户已完成实名身份认证"}],"followers_count":226072},"promotion":false,"answers":null,"view_count":403099,"created_at":1575376356000,"link_stock_desc":null,"link_stock_symbol":null,"strategy_id":0,"feedback":null,"tag":null,"card":null,"quote_cards":null,"source":"雪球","retweeted_status":null,"like_count":37,"liked":false,"mode":0,"symbol_id":null,"score":null,"reply_user_images":null,"reply_user_count":0,"offer":null,"user_id":1062883669,"longTextForIOS":false}',
   'id': 20358196,
   'original_status': None}],
 'next_id': 20358195,
 'next_max_id': 20358196,
 'tip': None}

模拟登录

• 为什么需要进行模拟登录：
  • 有些数据不经过登录是访问不到
• 验证码的处理:软件ID： 899370
  • 基于线上的打码平台来实现
    • 云打码：http://www.yundama.com/about.html
    • 超级鹰：http://www.chaojiying.com/about.html
  • 超级鹰的使用流程：
    • 基于用户中心的身份进行注册
    • 用户中心的身份进行登录：
      • 充值
      • 创建一个软件：软件ID-》生成软件ID
      • 下载示例代码：开发文档-》Python-》下载

import requests
from hashlib import md5

class Chaojiying_Client(object):

    def __init__(self, username, password, soft_id):
        self.username = username
        password =  password.encode('utf8')
        self.password = md5(password).hexdigest()
        self.soft_id = soft_id
        self.base_params = {
            'user': self.username,
            'pass2': self.password,
            'softid': self.soft_id,
        }
        self.headers = {
            'Connection': 'Keep-Alive',
            'User-Agent': 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)',
        }

    def PostPic(self, im, codetype):
        """
        im: 图片字节
        codetype: 题目类型 参考 http://www.chaojiying.com/price.html
        """
        params = {
            'codetype': codetype,
        }
        params.update(self.base_params)
        files = {'userfile': ('ccc.jpg', im)}
        r = requests.post('http://upload.chaojiying.net/Upload/Processing.php', data=params, files=files, headers=self.headers)
        return r.json()

    def ReportError(self, im_id):
        """
        im_id:报错题目的图片ID
        """
        params = {
            'id': im_id,
        }
        params.update(self.base_params)
        r = requests.post('http://upload.chaojiying.net/Upload/ReportError.php', data=params, headers=self.headers)
        return r.json()

#执行代码
chaojiying = Chaojiying_Client('bobo328410948', 'bobo328410948', '899370')
im = open('a.jpg', 'rb').read()
print chaojiying.PostPic(im, 1902)

• 使用打码平台识别古诗文网中的验证码图片

#封装一个识别验证码的函数
def transformCode(imgPath,imgType):
    chaojiying = Chaojiying_Client('bobo328410948', 'bobo328410948', '899370')
    im = open(imgPath, 'rb').read()
    return chaojiying.PostPic(im, imgType)['pic_str']

url = 'https://so.gushiwen.org/user/login.aspx?from=http://so.gushiwen.org/user/collect.aspx'
page_text = requests.get(url,headers=headers).text
tree = etree.HTML(page_text)
#解析到了验证码图片的地址
code_img_src = 'https://so.gushiwen.org'+tree.xpath('//*[@id="imgCode"]/@src')[0]
img_data = requests.get(code_img_src,headers=headers).content
with open('./code.jpg','wb') as fp:
    fp.write(img_data)
transformCode('./code.jpg',1902)

'uzzf'

• 模拟登录
  • 对点击登录按钮对应的url进行请求发送
  • 动态变化的请求参数：
    • 一般都会被隐藏在前台页面中
    • 基于抓包工具对请求参数的名称进行全局搜索

#处理cookie
session = requests.Session()

#获取且识别验证码
url = 'https://so.gushiwen.org/user/login.aspx?from=http://so.gushiwen.org/user/collect.aspx'
page_text = session.get(url,headers=headers).text
tree = etree.HTML(page_text)
#解析到了验证码图片的地址
code_img_src = 'https://so.gushiwen.org'+tree.xpath('//*[@id="imgCode"]/@src')[0]

#解析出模拟登录请求中动态变化的两个请求参数的之
__VIEWSTATE = tree.xpath('//*[@id="__VIEWSTATE"]/@value')[0]
__VIEWSTATEGENERATOR = tree.xpath('//*[@id="__VIEWSTATEGENERATOR"]/@value')[0]
#对图片进行请求时捕获cookie
img_data = session.get(code_img_src,headers=headers).content
with open('./code.jpg','wb') as fp:
    fp.write(img_data)
#验证码对应的文本数据
code_text = transformCode('./code.jpg',1902)
print(code_text)

#对于登录按钮的点击进行请求发送
login_url = 'https://so.gushiwen.org/user/login.aspx?from=http%3a%2f%2fso.gushiwen.org%2fuser%2fcollect.aspx'
data = {
    '__VIEWSTATE': __VIEWSTATE,
    '__VIEWSTATEGENERATOR': __VIEWSTATEGENERATOR,
    'from': 'http://so.gushiwen.org/user/collect.aspx',
    'email': 'www.zhangbowudi@qq.com',
    'pwd': 'bobo328410948',
    'code': code_text,
    'denglu': '登录',
}
login_page_text = session.post(url=login_url,headers=headers,data=data).text
with open('./gushiwen.html','w',encoding='utf-8') as fp:
    fp.write(login_page_text)

ivvy

• 总结：
  • 验证码
  • 动态变化的请求参数
  • cookie
  • 代理
  • UA
  • robots