使用tidylib解决不规则网页问题

windows 下安装tidylib

先去https://pypi.python.org/pypi/pytidylib

下载pytidylib

跑官方提供的测试用例报错

ition 0: unexpected end of data
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 314, in 'calling callback function'
  File "D:Python27libsite-packages	idylibsink.py", line 79, in put_byte
    write_func(byte.decode('utf-8'))
  File "D:Python27libencodingsutf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xb7 in position 0: invalid start byte

解决办法：

去http://www.paehl.com/open_source/?HTML_Tidy_for_Windows 下载LIBTIDY DLL

将dll解压到python安装目录下的Scripts文件夹即可

解决xpath提取不规则网页的问题

在数据采集时，处理不规范的HTML页面是件令人头疼的事。因为不规范的HTML页面往往会引起xpath解析失败，造成得不到正确的数据。 
这里引入tidylib,实例采用抓取慧聪网为例子,使用示例如下,

from tidylib import tidy_document
import requests
from lxml import etree

url = "https://s.hc360.com/?w=%BF%D5%B5%F7"
s = requests.Session()
# proxies={
#     "https":"https://192.168.43.1:1800"
# }
headers = {
    "Host": "s.hc360.com",
    "user-agent": "Mozilla/5.0 (X11; Linux x86_64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko)"
                  " Chrome/67.0.3396.87 Safari/537.36",
    "cookies": "hcsearchurlport=1; visitid_time=2018-8-27%2011%3A0%3A1; hc360visitid=C81E205C97B000016A91177717E91DA7; hc360first_time=2018-08-27; hcbrowserid=C81E205CA0100001BD5AC09D4D99F260; hckIndex=C81E205CA0A00001613715F0773DE730; topmatchkey=; Hm_lvt_1437b8f613f9fcba581e33d8d178e1f5=1535338807; hccordet=00; hcpreurl=; hclastsearchkeyword=%u7A7A%u8C03; Hm_lvt_e1e386be074a459371b2832363c0d7e7=1535338809; hc360sessionid=C81E205EE6000001B74C508019802980; hc360sessionid=C81E205EE6000001B74C508019802980; hc360firstvisittime=1535338811032; hc360firstvisittime=1535338811032; _ga=GA1.2.1558106052.1535338807; _gid=GA1.2.1254508720.1535338856; hc360analyid=C81E24E7EC600001801D413B1900ED60; hc360analycopyid=C81E24E7EC6000016E7E7D601F0098B0; Hm_lpvt_1437b8f613f9fcba581e33d8d178e1f5=1535345882;"
               " Hm_lpvt_e1e386be074a459371b2832363c0d7e7=1535345884; hc5minbeat=1535346223259"

}
response = s.get(url, headers=headers).text
response, errors = tidy_document(response)
res = etree.HTML(response)
items = res.xpath('//div/ul/li[contains(@class,"grid-list")]')
print(response)
print(len(items))
for item in items:
    d = dict()
    d['name'] = item.xpath('.//div[2]/dl/dd/p/a/text()')
    d['title'] = item.xpath('.//div[@class="NewItem"]/div/a/@title')
    d['price'] = item.xpath('.//div[2]/dl/dt/span/text()')
    d['bcid'] = item.xpath('.//div[@class="NewItem"]/@data-bcid')
    d['username'] =item.xpath('.//div[@class="NewItem"]/@data-username')
    d['tel'] =item.xpath('./@data-telphone')
    d['businid'] =item.xpath('./@data-businid')
    d['sellerproviderid'] =item.xpath('./@data-sellerproviderid')
    d['supcatid'] =item.xpath('./@data-supcatid')
    d['main_product'] =item.xpath('.//div[@class="NewItem"]/@data-obj')
    print(d)

可以匹配导数据

{'businid': ['668378345'], 'username': ['yzhh'], 'sellerproviderid': ['100020178332'], 'bcid': ['668378345'], 'tel': ['13801459353'], 'price': ['2.36万/套'], 'supcatid': ['051009003'], 'title': ['HAIER/海尔家用中央空调智尊·MXS系列一拖四套餐/室外机5匹125-4-A'], 'name': ['青岛海尔空调电子'], 'main_product': ['空调,:0']}
{'businid': ['416147806'], 'username': ['yhjddmc'], 'sellerproviderid': ['100018953096'], 'bcid': ['416147806'], 'tel': ['18562142856'], 'price': ['450.00/台'], 'supcatid': ['012034003001'], 'title': ['安徽优质供应风机盘管|2匹壁挂式远博牌风机盘管|德州远博空调'], 'name': ['德州远博空调有限'], 'main_product': ['风机盘管机组,水源热泵机组,地源热泵机组,恒温恒湿机组 ,风冷冷（热）水机组,水冷冷（热）水机组,卡式风机盘管,卧式明装风机盘管,卧室暗装风机盘管 ,立式明装风机盘管,空气处理机组,组合式空调机组 ,立式暗装风机盘管,风机盘管,嵌入式风机盘管:0']}
{'businid': ['680346169'], 'username': ['hisurp'], 'sellerproviderid': ['100017484813'], 'bcid': ['680346169'], 'tel': ['18968377151'], 'price': ['5000.00/台'], 'supcatid': ['051003005'], 'title': ['GLONTER/吉霖特射流机组 车间降温空调 工厂降空调 工业车间空调 高大空间空调'], 'name': ['浙江吉霖特制冷设'], 'main_product': ['精密机房空调,特种定制空调,高温空调,屋顶式空调机组,直膨式空调机组,工业空调,风冷热泵机组,集装箱空调,洁净式空调机组,恒温恒湿机组,全新风空调机组,帐篷空调,组合式空调机组,医用净化风机盘管,电气室专用空调,粮库空调,三联供空调,冷凝热回收排风机组,基站空调,电柜空调:0']}
......