Python爬虫相关技巧

get请求

kv = {
    'Cookie': 'ccpassport=ec081bd592c086d4057c3442b43b7998; wzwsconfirm=52277a99b139398795c925c264b5cf54; wzwstemplate=OQ==; wzwschallenge=-1;
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36',
     'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8'}
requests.adapters.DEFAULT_RETRIES = 5
requests.session().keep_alive = False
r = requests.get(url, headers=kv, timeout=60)
r.raise_for_status()
r.encoding = r.apparent_encoding

设置重连次数：requests.adapters.DEFAULT_RETRIES
设置连接活跃状态: requests.session().keep_alive = False
添加参数，params={}
查看http返回状态码：r.raise_for_status()
设置返回数据的编码：r.encoding
获取返回数据的文本：r.text; 若返回数据为json，可通过r.json()获取得到

获取Post请求

r = requests.post(url, data={}, headers=kv, timeout=60)

与get请求类似。注意，post请求参数名称为data

两者更具体的用法见：requests快速上手、requests高级用法、requests开发接口

Selenium 获取网站截图

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

fire_fox_user_agent = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0"
dcap = dict(DesiredCapabilities.PHANTOMJS)
dcap["phantomjs.page.settings.userAgent"] = fire_fox_user_agent
brower = webdriver.PhantomJS(desired_capabilities=dcap)
brower.set_page_load_timeout(180)
brower.get(url)
brower.maximize_window()
path = 'my.jpg'
brower.save_screenshot(path)
brower.close()

更多用法见：Python+Selenium WebDriver API：浏览器及元素的常用函数及变量整理总结、 Selenium API文件

将字典的json形式存入文件

import json
fp = open('my.json', 'a+', encoding='utf-8')
json.dump(dict, fp, ensure_ascii=False)
fp.close()

此外，json.dumps(dict)可将字典转为字符串

加载读取Json文件

import json
fp=open('my.json', 'r', encoding="utf-8")
dict = json.load(fp)
fp.close()

此外，json.loads()可将字符串转为字典，且必须是'{"xx":"c","f":"v"}'这种形式，即外面是单引号，里面是双引号，反之报错。

 下载文件

f = open("my.pdf",'wb') 
f.write(requests.get(url).content)
f.close()

日志

import logging

logging.basicConfig(filename= 'crawlLog.log',
                        level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")

logging.info("info")
logging.error("error ")

更具体的用法见：Python之日志处理（logging模块）

定时

import schedule

def job():
    print("I'm working...")

schedule.every(10).minutes.do(job)
schedule.every().hour.do(job)
schedule.every().day.at("10:30").do(job)
schedule.every().monday.do(job)
schedule.every().wednesday.at("13:15").do(job)

while True:
    schedule.run_pending()

schedule API见：https://schedule.readthedocs.io/en/stable/api.html

另，爬虫加密：简谈-Python爬虫破解JS加密的Cookie