- scrapy环境的安装
a. pip3 install wheel
b. 下载twisted http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
c. 进入下载目录，执行 pip3 install Twisted‑17.1.0‑cp35‑cp35m‑win_amd64.whl
d. pip3 install pywin32
e. pip3 install scrapy
- scrapy使用
- 1.创建一个工程：scrapy startproject ProName （注 startproject 中间没有空格）
- 2.cd ProName
- 3.创建爬虫文件：scrapy genspider first www.xxx.com
- 4.执行：（allowed_domains 注释掉）
- settings.py:
- 不遵从rbotes协议
- 进行UA伪装
- 指定日志等级：LOG_LEVEL = ‘ERROR’
scrapy crawl spiderName
- 持久化存储
- 基于终端指令：
- 前提：只可以将parse方法的返回值进行本地文件的持久化存储
- 指令：scrapy crawl spiderName -o filePath
- 基于管道：
- 编码流程：
1.数据解析
2.需要在item类中定义相关的属性（存储解析到的数据）(在 items.py 中添加属性)
3.将解析到的数据存储或者封装到一个item类型的对象中（在 主文件里 实例化item 封装数据） （注是item['name'] 不是item.name）
4.将item对象提交到管道中(yield item)
5.在管道中需要接收item，且将item对象中的数据进行任意形式的持久化操作（在piplines.py里添加
）
6.在配置文件中开启管道（settings 里打开 ITEM_PIPELINES）
5.scrapy如何实现持久化存储？
- 1.数据解析
- 2.封装item类（定义属性）
- 3.将解析的数据存储到item对象
- 4.将item提交到管道
- 5.在管道中接受item且对其进行任意i形式的持久化存储
- 6.在配置文件中开启管道
6.简述基于移动端数据爬取的配置流程？
- 下载fiddler
- 配置fiddler：
- 配置端口
- allow remote xxx
- 在手机端下载安装证书
- 手机和fiddler的pc端处于同一个网段（pc分享一个热点手机连接）
- 在手机浏览器中访问ip：port下载证书
- 在手机中设置网络代理：代理信息都是来源于fiddler所在的pc
scrapy管道的细节处理
- 数据的爬取和持久化存储，将同一组数据分别存储到不同的平台
- 一个管道类负责将数据存储到某一个载体或者平台中
- 爬虫文件提交的item只会提交给第一个被执行的管道类
- 在管道类的process_item中的return item表示的含义就是将当前管道类接收到的item传递给下一个即将被执行的管道类
- 注意：爬虫类和管道类进行数据交互的形式
- yild item：只可以交互item类型的对象
- spider参数：可以交互任意形式的数据
- 全站数据的爬取
- 基于手动请求发送实现
- 实现全站数据爬取
- 实现深度爬取
- 手动请求发送：
- yield scrapy.Request(url,callback)
- yield scrapy.FormRequest(url,formdata,callback)
- post请求的发送和cookie的处理
- cookie的处理是scrapy自动封装好
- start_urls自动请求发送的实现原理：
#是父类的一个请求发送的封装
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url=url,callback=self.parse)
- 深度爬取
- 手动请求发送
- 请求传参：持久化存储,将不同的回调中解析的数据存储到同一个item对象。请求传参传递的就是item对象。
- 使用场景：如果使用scrapy爬取的数据没有在同一张页面中
- 传递方式：将传递数据封装到meta字典中，meta传递个了callback
yield scrapy.Request(url,callback,meta)
- 接收：
在callback指定的回调函数中使用response进行接收：
- item = response.meta['key']
- 五大核心组件
- 引擎(Scrapy)
用来处理整个系统的数据流处理, 触发事务(框架核心)
- 调度器(Scheduler)
用来接受引擎发过来的请求, 压入队列中, 并在引擎再次请求的时候返回. 可以想像成一个URL（抓取网页的网址或者说是链接）的优先队列, 由它来决定下一个要抓取的网址是什么, 同时去除重复的网址
- 下载器(Downloader)
用于下载网页内容, 并将网页内容返回给蜘蛛(Scrapy下载器是建立在twisted这个高效的异步模型上的)
- 爬虫(Spiders)
爬虫是主要干活的, 用于从特定的网页中提取自己需要的信息, 即所谓的实体(Item)。用户也可以从中提取出链接,让Scrapy继续抓取下一个页面
- 项目管道(Pipeline)
负责处理爬虫从网页中抽取的实体，主要的功能是持久化实体、验证实体的有效性、清除不需要的信息。当页面被爬虫解析后，将被发送到项目管道，并经过几个特定的次序处理数据。
- 如何提升scrapy爬取数据的效率
增加并发：
默认scrapy开启的并发线程为32个，可以适当进行增加。在settings配置文件中修改CONCURRENT_REQUESTS = 100值为100,并发设置成了为100。
降低日志级别：
在运行scrapy时，会有大量日志信息的输出，为了减少CPU的使用率。可以设置log输出信息为INFO或者ERROR即可。在配置文件中编写：LOG_LEVEL = ‘INFO’
禁止cookie：
如果不是真的需要cookie，则在scrapy爬取数据时可以禁止cookie从而减少CPU的使用率，提升爬取效率。在配置文件中编写：COOKIES_ENABLED = False
禁止重试：
对失败的HTTP进行重新请求（重试）会减慢爬取速度，因此可以禁止重试。在配置文件中编写：RETRY_ENABLED = False
减少下载超时：
如果对一个非常慢的链接进行爬取，减少下载超时可以能让卡住的链接快速被放弃，从而提升效率。在配置文件中进行编写：DOWNLOAD_TIMEOUT = 10 超时时间为10s
BOSS直聘深度爬取
import scrapy from bossDeepPro.items import BossdeepproItem class BossSpider(scrapy.Spider): name = 'boss' # allowed_domains = ['www.xxx.com'] start_urls = ['https://www.zhipin.com/job_detail/?query=python%E5%BC%80%E5%8F%91&city=101010100&industry=&position='] # 通用的url模板（不可变） url = 'https://www.zhipin.com/c101010100/?query=python开发&page=%d' page = 2 def parse(self, response): print('正在爬取第{}页的数据'.format(self.page)) li_list = response.xpath('//*[@id="main"]/div/div[3]/ul/li | //*[@id="main"]/div/div[2]/ul/li') for li in li_list: job_title = li.xpath('.//div[@class="info-primary"]/h3/a/div[1]/text()').extract_first() salary = li.xpath('.//div[@class="info-primary"]/h3/a/span/text()').extract_first() #实例化item对象：对象必须要让parse和parse_detail共享 item = BossdeepproItem() item['job_title'] = job_title item['salary'] = salary detail_url = 'https://www.zhipin.com'+li.xpath('.//div[@class="info-primary"]/h3/a/@href').extract_first() #对详情页的url发起手动请求 yield scrapy.Request(url=detail_url,callback=self.parse_detail,meta={'item':item}) #(多个管道间的数据传输) if self.page <= 5: # 对其他页码进行手动请求的发送 new_url = format(self.url % self.page) print(new_url) self.page += 1 # 手动请求发送 # callback进行数据解析 yield scrapy.Request(url=new_url, callback=self.parse) #解析岗位描述 def parse_detail(self,response): item = response.meta['item'] job_desc = response.xpath('//*[@id="main"]/div[3]/div/div[2]/div[2]/div[1]/div//text()').extract() job_desc = ''.join(job_desc) item['job_desc'] = job_desc yield item
import scrapy class BossdeepproItem(scrapy.Item): # define the fields for your item here like: job_title = scrapy.Field() job_desc = scrapy.Field() salary = scrapy.Field()
# -*- coding: utf-8 -*- # Define here the models for your spider middleware # # See documentation in: # https://docs.scrapy.org/en/latest/topics/spider-middleware.html from scrapy import signals class BossproSpiderMiddleware(object): # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the spider middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_spider_input(self, response, spider): # Called for each response that goes through the spider # middleware and into the spider. # Should return None or raise an exception. return None def process_spider_output(self, response, result, spider): # Called with the results returned from the Spider, after # it has processed the response. # Must return an iterable of Request, dict or Item objects. for i in result: yield i def process_spider_exception(self, response, exception, spider): # Called when a spider or process_spider_input() method # (from other spider middleware) raises an exception. # Should return either None or an iterable of Request, dict # or Item objects. pass def process_start_requests(self, start_requests, spider): # Called with the start requests of the spider, and works # similarly to the process_spider_output() method, except # that it doesn’t have a response associated. # Must return only requests (not items). for r in start_requests: yield r def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name) class BossproDownloaderMiddleware(object): # Not all methods need to be defined. If a method is not defined, # scrapy acts as if the downloader middleware does not modify the # passed objects. @classmethod def from_crawler(cls, crawler): # This method is used by Scrapy to create your spiders. s = cls() crawler.signals.connect(s.spider_opened, signal=signals.spider_opened) return s def process_request(self, request, spider): # Called for each request that goes through the downloader # middleware. # Must either: # - return None: continue processing this request # - or return a Response object # - or return a Request object # - or raise IgnoreRequest: process_exception() methods of # installed downloader middleware will be called return None def process_response(self, request, response, spider): # Called with the response returned from the downloader. # Must either; # - return a Response object # - return a Request object # - or raise IgnoreRequest return response def process_exception(self, request, exception, spider): # Called when a download handler or a process_request() # (from other downloader middleware) raises an exception. # Must either: # - return None: continue processing this exception # - return a Response object: stops process_exception() chain # - return a Request object: stops process_exception() chain pass def spider_opened(self, spider): spider.logger.info('Spider opened: %s' % spider.name)
class BossdeepproPipeline(object): def process_item(self, item, spider): print(item) return item
# -*- coding: utf-8 -*- # Scrapy settings for bossDeepPro project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://docs.scrapy.org/en/latest/topics/settings.html # https://docs.scrapy.org/en/latest/topics/downloader-middleware.html # https://docs.scrapy.org/en/latest/topics/spider-middleware.html BOT_NAME = 'bossDeepPro' SPIDER_MODULES = ['bossDeepPro.spiders'] NEWSPIDER_MODULE = 'bossDeepPro.spiders' USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.87 Safari/537.36' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'bossDeepPro (+http://www.yourdomain.com)' # Obey robots.txt rules ROBOTSTXT_OBEY = False LOG_LEVEL = 'ERROR' # LOG_FILE = 'log.txt' # Configure maximum concurrent requests performed by Scrapy (default: 16) # CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) #COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', # 'Accept-Language': 'en', #} # Enable or disable spider middlewares # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # 'bossDeepPro.middlewares.BossdeepproSpiderMiddleware': 543, #} # Enable or disable downloader middlewares # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html #DOWNLOADER_MIDDLEWARES = { # 'bossDeepPro.middlewares.BossdeepproDownloaderMiddleware': 543, #} # Enable or disable extensions # See https://docs.scrapy.org/en/latest/topics/extensions.html #EXTENSIONS = { # 'scrapy.extensions.telnet.TelnetConsole': None, #} # Configure item pipelines # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html ITEM_PIPELINES = { 'bossDeepPro.pipelines.BossdeepproPipeline': 300, } # Enable and configure the AutoThrottle extension (disabled by default) # See https://docs.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = 'httpcache' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'