【Python】 requests 爬取博客园内容AttributeError: 'NoneType' object has no attribute 'xpath'

本篇文章主要介绍requests获取网页内容出现 'NoneType' object has no attribute 'xpath' 异常的解决思路

下面是出错的代码：

import requests
from lxml import etree
response = requests.get('https://blog.csdn.net/it_xf?viewmode=contents')
etree_html = etree.HTML(response.text)
content = etree_html.xpath('//*[@id="mainBox"]/main/div[2]/div[1]/h4/a/text()')

for each in content:
    replace = each.replace('
', '').replace(' ', '')
    if replace == '
' or replace == '':
        continue
    else:
        print(replace)　　

1、错误分析

获取到的html.text 为 空字符串；所以下面抛出异常NoneType 

原因是请求Get 需要增加 headers来解决反扒；模拟浏览器请求来获取数据；

2、解决办法

首先找到需要的headers，headers 如何寻找？看下图的标记：

然后直接把上面的headers复制出来放到代码中进行改造；

改造后的代码如下：

import requests
from lxml import etree
headers = {
    'user-agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) '
                 'AppleWebKit/537.36 (KHTML, like Gecko) '
                 'Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0'
}
response = requests.get('https://blog.csdn.net/it_xf?viewmode=contents', headers = headers)
etree_html = etree.HTML(response.text)
content = etree_html.xpath('//*[@id="mainBox"]/main/div[2]/div[1]/h4/a/text()')

for each in content:
    replace = each.replace('
', '').replace(' ', '')
    if replace == '
' or replace == '':
        continue
    else:
        print(replace)