Scrapy框架: 通用爬虫之CrawlSpider

步骤01: 创建爬虫项目

scrapy startproject quotes

步骤02: 创建爬虫模版

scrapy genspider -t quotes quotes.toscrape.com

步骤03: 配置爬虫文件quotes.py

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor


class Quotes(CrawlSpider):
	# 爬虫名称
    name = "get_quotes"
    allow_domain = ['quotes.toscrape.com']
    start_urls = ['http://quotes.toscrape.com/']

# 设定规则
    rules = (
        # 对于quotes内容页URL，调用parse_quotes处理，
      		# 并以此规则跟进获取的链接
        Rule(LinkExtractor(allow=r'/page/d+'), callback='parse_quotes', follow=True),
      		# 对于author内容页URL，调用parse_author处理，提取数据
        Rule(LinkExtractor(allow=r'/author/w+'), callback='parse_author')
    )

# 提取内容页数据方法
    def parse_quotes(self, response):
        for quote in response.css(".quote"):
            yield {'content': quote.css('.text::text').extract_first(),
                   'author': quote.css('.author::text').extract_first(),
                   'tags': quote.css('.tag::text').extract()
                   }
	# 获取作者数据方法

    def parse_author(self, response):
        name = response.css('.author-title::text').extract_first()
        author_born_date = response.css('.author-born-date::text').extract_first()
        author_bron_location = response.css('.author-born-location::text').extract_first()
        author_description = response.css('.author-description::text').extract_first()

        return ({'name': name,
                 'author_bron_date': author_born_date,
                 'author_bron_location': author_bron_location,
                 'author_description': author_description
                 })

步骤04: 运行爬虫

scrapy crawl quotes