爬取知乎热榜标题和连接 （python，requests，xpath）

用python爬取知乎的热榜，获取标题和链接。

环境和方法：ubantu16.04、python3、requests、xpath

 

1.用浏览器打开知乎，并登录

 

2.获取cookie和User—Agent

 

3.上代码

import requests
from lxml import etree

def get_html(url):
    headers={
            'Cookie':'你的Cookie',
            #'Host':'www.zhihu.com',
            'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
            }

    r=requests.get(url,headers=headers)

    if r.status_code==200:
        deal_content(r.text)

def deal_content(r):
    html = etree.HTML(r)
    title_list = html.xpath('//*[@id="TopstoryContent"]/div/section/div[2]/a/h2')
    link_list = html.xpath('//*[@id="TopstoryContent"]/div/section/div[2]/a/@href')
    for i in range(0,len(title_list)):
        print(title_list[i].text)
        print(link_list[i])
        with open("zhihu.txt",'a') as f:
            f.write(title_list[i].text+'
')
            f.write('	链接为：'+link_list[i]+'
')
            f.write('*'*50+'
')

def main():
    url='https://www.zhihu.com/hot'
    get_html(url)

main()

4.爬取结果