Scrapy项目_阳光热线问政平台

目的：

爬取阳光热线问政平台问题中每个帖子的标题、详情URL、详情内容、图片以及发布时间

 

步骤：

1、创建爬虫项目

scrapy startproject yangguang
cd yangguang
scrapy genspider yangguang sun0769.com

2、设置item.py文件

import scrapy


class YangguangItem(scrapy.Item):
    # 每条帖子的标题
    title = scrapy.Field()
    # 帖子链接
    href = scrapy.Field()
    # 发布日期
    publish_time = scrapy.Field()
    # 详情内容
    content = scrapy.Field()
    # 详情图片
    content_img = scrapy.Field()    

 3、编写爬虫文件

import scrapy
from yangguang.items import YangguangItem


class YgSpider(scrapy.Spider):
    name = 'yg'
    allowed_domains = ['sun0769.com']
    start_urls = ['http://wz.sun0769.com/political/index/politicsNewest?id=1&page=1']

    page = 1
    url = "http://wz.sun0769.com/political/index/politicsNewest?id=1&page= {}"

    def parse(self, response):
        # 分组
        li_list = response.xpath("//ul[@class='title-state-ul']/li")
        for li in li_list:
            item = YangguangItem()
            item["title"] = li.xpath("./span[@class='state3']/a/text()").extract_first()
            item["href"] = "http://wz.sun0769.com" + li.xpath("./span[@class='state3']/a/@href").extract_first()
            item["publish_time"] = li.xpath("./span[@class='state5 ']/text()").extract_first()

            yield scrapy.Request(
                item["href"],
                callback=self.parse_detail,
                meta={
                    "item": item,
                    "proxy": "http://171.12.221.51:9999"
                }
            )
        # 翻页
        if self.page < 10:
            self.page += 1
            next_url = self.url.format(self.page)

            yield scrapy.Request(next_url, callback=self.parse, meta={"proxy": "http://123.163.118.71:9999"})

    def parse_detail(self, response):  # 处理详情页
        item = response.meta["item"]
        item["content"] = response.xpath("//div[@class='details-box']/pre/text()").extract_first()
        item["content_img"] = response.xpath("//div[@class='clear details-img-list Picture-img']/img/@src").extract()
        yield item

4、测试

scrapy crawl yg