• CrawlSpider一直无法进入回调函数


    CrawlSpider一直无法进入回调函数的

    碰到的两个坑:

    1、正则表达式写的不对

    start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1&page=1']
    
    link = LinkExtractor(allow=r'id=1&page=d+')
    


    2、回调函数中的xpath路径写的不对


    附本次代码:
    import scrapy
    from scrapy.linkextractors import LinkExtractor
    from scrapy.spiders import CrawlSpider, Rule
    from sunCrawlSpider.items import SuncrawlspiderItem
    from sunCrawlSpider.items import Detail_item
    
    
    class SunSpider(CrawlSpider):
        name = 'sun'
        # allowed_domains = ['www.xxx.com']
        start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1&page=1']
    
        link = LinkExtractor(allow=r'id=1&page=d+')
    
        link_detail = LinkExtractor(allow=r'political/politics/index?id=d+')
    
        rules = (
            Rule(link, callback='parse_item', follow=False),
            Rule(link_detail, callback='parse_detail', follow=False),
        )
    
        def parse_item(self, response):
            li_list = response.xpath('/html/body/div[2]/div[3]/ul[2]/li')
            for li in li_list:
                num = li.xpath('./span[1]/text()').extract_first()
                title = li.xpath('./span[3]/a/text()').extract_first()
                # url = 'http://wz.sun0769.com/' + li.xpath('./span[3]/a/@href').extract_first()
                item = SuncrawlspiderItem()
                item['title'] = title
                item['num'] = num
    
                yield item
    
        def parse_detail(self, response):
            content = response.xpath('/html/body/div[3]/div[2]/div[2]/div[2]/pre/text()').extract_first()
            num = response.xpath('/html/body/div[3]/div[2]/div[2]/div[1]/span[4]/text()').extract_first()
            num = num.split(':')[-1]
            item = Detail_item()
            item['num'] = num
            item['content'] = content
    
            yield item
    

      




  • 相关阅读:
    delphi 使用条件编译指令
    [转] 编程之道 二
    delphi中XLSReadWrite控件的使用(1)简介
    delphi中XLSReadWrite控件的使用(2)delphi XE下安装
    墙纸自动换1.4算法分析
    Delphi中设置屏幕分辨率
    delphistringgrid另类自动向下滚屏
    【Hex 格式文件操作】一、intel hex格式文件说明
    INTEL hex文件格式
    [转]编程之道 一
  • 原文地址:https://www.cnblogs.com/wlj-axia/p/13378474.html
Copyright © 2020-2023  润新知