# nvshens图片批量下载爬虫1.01 # 原先版本在遇到网络故障时回下载不全，这回更改了模式使得下载不成就重新下载，直到全部下载完毕 from bs4 import BeautifulSoup import requests import time import urllib.request user_agent='Mozilla/4.0 (compatible;MEIE 5.5;windows NT)' headers={'User-Agent':user_agent} # 找到的图片 pictures=[] # 不断追逐，直到结束 def crawl(url): print("爬取页面"+url); try: rsp=requests.get(url,headers=headers) soup= BeautifulSoup(rsp.text,'html.parser',from_encoding='utf-8') nextUrl="none"; for divs in soup.find_all(class_="gallery_wrapper"): # 把找到的图片放到数组里去 for img in divs.find_all('img'): print(img.get("src")) pictures.append(img.get("src")) #找下一页 for link in divs.find_all('a',class_='a1'): if link.string=='下一页' and link.get("href").find('.html')!=-1: nextUrl='https://www.nvshens.com'+link.get("href"); #if len(pictures)>10: # downloadPics() #time.sleep(5) if nextUrl!="none": print("前往下一页"); crawl(nextUrl) else: print('爬取结束，开始下载...') downloadPics() print('下载结束，nvshens图片批量下载爬虫工作完毕.') except Exception as e: print("发生异常。重新爬行")# 不管怎么出现的异常，就让它一直爬到底 crawl(url) # 下载图片到本地 def downloadPics(): while(len(pictures)>0): pic=pictures.pop() name=pic.split('/')[-1] try: rsp=urllib.request.urlopen(pic) img=rsp.read() with open(name,'wb') as f: f.write(img) print('图片'+pic+'下载完成') except Exception as e: print('图片'+pic+'下载异常,塞回重试') pictures.append(pic); # Kickoff Start # https://www.nvshens.com/g/22268/ crawl('https://www.nvshens.com/g/22210/')
输出：
C:Usershorn1Desktoppython7>python downloadall.py 爬取页面https://www.nvshens.com/g/22210/ C:Usershorn1AppDataLocalProgramsPythonPython36libsite-packagess4__init__.py:146: UserWarning: You provided Unicode markup but also provided a value for from_encoding. Your from_encoding will be ignored. warnings.warn("You provided Unicode markup but also provided a value for from_encoding. Your from_encoding will be ignored.") https://img.onvshen.com:85/gallery/23789/22210/s/0.jpg https://img.onvshen.com:85/gallery/23789/22210/s/001.jpg https://img.onvshen.com:85/gallery/23789/22210/s/002.jpg https://img.onvshen.com:85/gallery/23789/22210/s/003.jpg https://img.onvshen.com:85/gallery/23789/22210/s/004.jpg 前往下一页 爬取页面https://www.nvshens.com/g/22210/2.html https://img.onvshen.com:85/gallery/23789/22210/s/005.jpg https://img.onvshen.com:85/gallery/23789/22210/s/006.jpg https://img.onvshen.com:85/gallery/23789/22210/s/007.jpg https://img.onvshen.com:85/gallery/23789/22210/s/008.jpg https://img.onvshen.com:85/gallery/23789/22210/s/009.jpg 前往下一页 爬取页面https://www.nvshens.com/g/22210/3.html https://img.onvshen.com:85/gallery/23789/22210/s/010.jpg https://img.onvshen.com:85/gallery/23789/22210/s/011.jpg https://img.onvshen.com:85/gallery/23789/22210/s/012.jpg https://img.onvshen.com:85/gallery/23789/22210/s/013.jpg https://img.onvshen.com:85/gallery/23789/22210/s/014.jpg 前往下一页 爬取页面https://www.nvshens.com/g/22210/4.html https://img.onvshen.com:85/gallery/23789/22210/s/015.jpg https://img.onvshen.com:85/gallery/23789/22210/s/016.jpg https://img.onvshen.com:85/gallery/23789/22210/s/017.jpg https://img.onvshen.com:85/gallery/23789/22210/s/018.jpg https://img.onvshen.com:85/gallery/23789/22210/s/019.jpg 前往下一页 爬取页面https://www.nvshens.com/g/22210/5.html https://img.onvshen.com:85/gallery/23789/22210/s/020.jpg https://img.onvshen.com:85/gallery/23789/22210/s/021.jpg https://img.onvshen.com:85/gallery/23789/22210/s/022.jpg https://img.onvshen.com:85/gallery/23789/22210/s/023.jpg https://img.onvshen.com:85/gallery/23789/22210/s/024.jpg 前往下一页 爬取页面https://www.nvshens.com/g/22210/6.html https://img.onvshen.com:85/gallery/23789/22210/s/025.jpg https://img.onvshen.com:85/gallery/23789/22210/s/026.jpg https://img.onvshen.com:85/gallery/23789/22210/s/027.jpg https://img.onvshen.com:85/gallery/23789/22210/s/028.jpg https://img.onvshen.com:85/gallery/23789/22210/s/029.jpg 前往下一页 爬取页面https://www.nvshens.com/g/22210/7.html https://img.onvshen.com:85/gallery/23789/22210/s/030.jpg https://img.onvshen.com:85/gallery/23789/22210/s/031.jpg https://img.onvshen.com:85/gallery/23789/22210/s/032.jpg https://img.onvshen.com:85/gallery/23789/22210/s/033.jpg https://img.onvshen.com:85/gallery/23789/22210/s/034.jpg 前往下一页 爬取页面https://www.nvshens.com/g/22210/8.html https://img.onvshen.com:85/gallery/23789/22210/s/035.jpg https://img.onvshen.com:85/gallery/23789/22210/s/036.jpg https://img.onvshen.com:85/gallery/23789/22210/s/037.jpg https://img.onvshen.com:85/gallery/23789/22210/s/038.jpg https://img.onvshen.com:85/gallery/23789/22210/s/039.jpg 爬取结束，开始下载... 图片https://img.onvshen.com:85/gallery/23789/22210/s/039.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/039.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/038.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/038.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/038.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/037.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/036.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/036.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/035.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/034.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/033.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/032.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/031.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/030.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/029.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/028.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/028.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/027.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/027.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/026.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/025.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/025.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/025.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/024.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/024.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/024.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/023.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/022.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/022.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/021.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/020.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/019.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/019.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/019.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/019.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/018.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/018.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/018.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/017.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/016.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/015.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/014.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/013.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/012.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/011.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/010.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/009.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/008.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/008.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/008.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/007.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/006.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/005.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/005.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/005.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/005.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/005.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/004.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/003.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/002.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/001.jpg下载完成 图片https://img.onvshen.com:85/gallery/23789/22210/s/0.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/0.jpg下载异常,塞回重试 图片https://img.onvshen.com:85/gallery/23789/22210/s/0.jpg下载完成 下载结束，nvshens图片批量下载爬虫工作完毕.
：）