pipelines.py
import json
class xxPipeline(object):
def __init__(self):
self.filename=open("xx.json","wb")
def process_item(self, item, spider):
jsontext=json.dumps(dict(item),ensure_ascii=False) + ",
"
self.filename.write(jsontext.encode("utf-8"))
return item
def close_spider(self,spider):
self.filename.close()
注意:须将它的类添加到 settings.py文件ITEM_PIPELINES 配置
ITEM_PIPELINES = {"mySpider.pipelines.xxPipeline":300
}
也可通过命令
scrapy crawl xx(爬虫名字) -o xx.json
scrapy crawl xx(爬虫名字) -o xx.csv
scrapy crawl xx(爬虫名字) -o xx.xml