Scrapy爬虫之豆瓣TOP250

最近学习了python的一个超级牛的库scrapy，写下一些心得。

初看的时候，看的是官方文档，讲的有些晦涩，有些地方也有模糊不清的地方，而且完整的中高级的用例不多，再由于版本更新的问题，原来的一些方法发生了一些改变，所以在博客园上找到一篇博客结合官方文档，爬出一个自己的scrapy，主要目标豆瓣电影top250，接下来上源码：

首先，在自己想要的目录下新建项目：

scrapy startproject douban

进入douban文件夹，看一下目录结构，其中result.txt文件是我的输出文件

接下来，修改douban文件夹下的items.py，将返回项整合在一个item中:

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class DoubanItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    # pass
    movie_name = scrapy.Field()
    movie_director = scrapy.Field()
    movie_editor = scrapy.Field()
    movie_roles = scrapy.Field()
    movie_style = scrapy.Field()
    movie_date = scrapy.Field()
    movie_long = scrapy.Field()

然后开始编写自己的爬虫程序，在spiders文件下新建douban_spider.py：

# -*- coding: utf-8 -*-
from scrapy.spiders import BaseSpider   # 这里新版使用spiders
from scrapy.selector import HtmlXPathSelector
from douban.items import DoubanItem
import scrapy
import re
import sys
reload(sys)
sys.setdefaultencoding("utf-8")  # 设定文件字符编码utf-8


class DoubanSpider(BaseSpider):
    """docstring for DoubanSpider"""
    name = "douban"    # scrapy爬虫名称
    allow_domains = ["movie.douban.com"]   # 允许域名
    # 开始检索的URL
    start_urls = ["http://movie.douban.com/top250" + "?start=" + str(yeshu * 25) + "&filter=&type=" for yeshu in range(0, 10)]

    # 回调函数
    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        movie_link = hxs.xpath('//div[@class="hd"]/a/@href').extract()
        # movie_next = hxs.xpath('//span[@class="next"]/a/@href').extract()
        # nextmo = movie_next[0]
        # if nextmo:
        #     nextmo = "http://movie.douban.com/top" + nextmo
        #     start_urls.append(nextmo)
        for link in movie_link:
            # 给出进入二级页面的请求，并使用下面的回调函数
            yield scrapy.Request(link, callback=self.parse_item)

    # 自己写的回调函数，用于处理二级页面
    def parse_item(self, response):
        item_has = HtmlXPathSelector(response)
        movie_name = item_has.xpath('//h1/span/text()').extract()
        movie_director = item_has.xpath('//a[@rel="
            v:directedBy"]/text()').extract()
        movie_editor = item_has.xpath('//div[@id="
            info"]/span[2]/span[@class="attrs"]/a/text()').extract()
        movie_roles = item_has.xpath('//a[@rel="
            v:starring"]/text()').extract()
        movie_style = item_has.xpath('//span[@property="
            v:genre"]/text()').extract()
        movie_date = item_has.xpath('//span[@property="
            v:initialReleaseDate"]/text()').extract()
        movie_long = item_has.xpath('//span[@property="
            v:runtime"]/text()').extract()
        item = DoubanItem()
        item['movie_name'] = movie_name
        item['movie_director'] = movie_director
        item['movie_editor'] = movie_editor
        item['movie_roles'] = movie_roles
        item['movie_style'] = movie_style
        item['movie_date'] = movie_date
        item['movie_long'] = movie_long
        yield item

最后更改douban下的pinelines.py文件，用于将爬到的数据存到文件中：

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
NUM = 1


class DoubanPipeline(object):

    # NUM = 1

    def process_item(self, item, spider):
        movie_name = item['movie_name']
        movie_director = item['movie_director']
        movie_editor = [line + '、' for line in item['movie_editor']]
        movie_roles = [line + '、' for line in item['movie_roles']]
        movie_style = [line + '、' for line in item['movie_style']]
        movie_date = [line + '、' for line in item['movie_date']]
        movie_long = item['movie_long']
        f = open("result.txt", "a")
        global NUM
        f.write(str(NUM))
        f.write("
片名：")
        NUM += 1
        print "NAME:", movie_name
        # for it in movie_name:
        #     f.write(it)
        # f.write("
")
        f.writelines(movie_name)
        f.write("
导演：")
        f.writelines(movie_director)
        f.write("
编剧：")
        f.writelines(movie_editor)
        f.write("
主角：")
        f.writelines(movie_roles)
        f.write("
类型：")
        f.writelines(movie_style)
        f.write("
上映时间：")
        f.writelines(movie_date)
        f.write("
影片时长：")
        f.writelines(movie_long)
        f.write("
")
        f.close()
        return item

最后的最后，别忘了修改douban下的settings.py中的ITEM_PIPELINES，将它设置成我们自己写的pineline，在默认情况下是被注释掉的：

ITEM_PIPELINES = {
   'douban.pipelines.DoubanPipeline': 300,
}

程序工作就完成了，接下来你就可以运行了，需要在程序根目录运行，即第一个douban下：

scrapy crawl douban

​