总览
今天我们就来小用一下Jsoup，从一个整体的角度来看一看爬虫
一个基本的爬虫框架包括：
- [x] 解析网页
- [x] 失败重试
- [x] 抓取内容保存至本地
- [x] 多线程抓取
分模块讲解
将上述基本框架的模块按逻辑顺序讲解，一步一步复现代码实现过程
- 失败重试
一个好的模块必然有异常捕捉和处理
在之前的内容中，我们提到过一个简单的异常处理，小伙伴还记得么
简易版
// 爬取的网址
val url = "https://www.zhihu.com/explore/recommendations"
// 加上TryCatch框架
Try(Jsoup.connect(url).get())match {
case Failure(e) =>
// 打印异常信息
println(e.getMessage)
case Success(doc:Document) =>
// 解析正常则返回Document，然后提取Document内所需信息
println(doc.body())
}
今天我们来在之上稍微丰富一下，把他包装的更健壮一点
丰富版
var count = 0 //解析网页时统计抓取数用
//用于记录总数，和失败次数
val sum, fail: AtomicInteger = new AtomicInteger(0)
//当出现异常时1s后重试,异常重复100次
def requestGetUrl(times:Int=100,delay:Long=1000) : Unit ={
Try(Jsoup.connect(Url).userAgent(get_agent()).get())match {
case Failure(e) =>{
if(times!=0){
println(e.getMessage) //打印错误信息
Thread.sleep(delay) //等待1s
fail.addAndGet(1) //失败次数+1
requestGetUrl(times-1,delay) //times-1后，重调方法
}else throw e
}
case Success(doc) =>
parseDoc(doc)
if (count==0){ // 解析网页时用来统计是否抓取为空
Thread.sleep(delay)
requestGetUrl(times-1,delay)
}
sum.addAndGet(1) //成功次数+1
}
}
- get_agent()说明
//自己设置一下user-agent，或者更好的是，可以从一系列的user-agent里随机挑出一个符合标准的使用
def get_agent()={
//模拟header的user-agent字段，返回一个随机的user-agent字典类型的键值对
val agents=Array("Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv,2.0.1) Gecko/20100101 Firefox/4.0.1",
"Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)",
"Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11")
val ran = new Random().nextInt(agents.length)
agents(ran)
}
- 解析网页
沿用上一篇我们写过的方法就可以
//解析Document
var count = 0
//用一个hashmap来保存住区的内容
val text = new ConcurrentHashMap[String,String]()
def parseDoc(doc:Document): Unit ={
// 解析正常则返回Document，然后提取Document内所需信息
val links = doc.select("div.zm-item") //选取class为"zm-item"的div
for (link<-links.asScala) { //遍历每一个这样的div
val title = link.select("h2").text() //选取div中的所有"h2"标签，并读取它的文本内容
val approve = link.select("div.zm-item-vote").text() //找到赞同的位置，选中它并读取它的文本内容
//逐层找到唯一识别的标签，然后选中（唯一识别很关键）
val author = link.select("div.answer-head").select("span.author-link-line").select("a").text()
val content = link.select("div.zh-summary.summary.clearfix").text() //多个class类型，直接加.就行，如.A.B.C
text.put(title,author+" "+approve+" "+content)
count+=1
}
count
}
- 抓取内容保存至本地
// 获取当前日期
def getNowDate(): String ={
new SimpleDateFormat("yyMMdd").format(new Date())
}
// 爬取内容写入文件
def output(zone:String): Unit ={
val writer = new PrintWriter(new File(getNowDate()+"_"+zone++".txt"))
for((title,value)<-text){
writer.println(title+value)
}
writer.flush()
writer.close()
}
抓取内容展示
- 多线程抓取
//多线程抓取
def concurrentCrawler(zone: String,maxPage:Int,threadNum:Int)={
var loopar = (1 to maxPage).par
loopar.tasksupport = new ForkJoinTaskSupport(new ForkJoinPool(threadNum))
loopar.foreach(x=>requestGetUrl())
output(zone)
}
- get_agent()补充说明及福利
def get_agent()={
//模拟header的user-agent字段，返回一个随机的user-agent字典类型的键值对
val agents=Array("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
"Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
"Mozilla/5.0 (Macintosh; U; Mac OS X Mach-O; en-US; rv:2.0a) Gecko/20040614 Firefox/3.0.0 ",
"Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9.0.3) Gecko/2008092414 Firefox/3.0.3",
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5",
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.14) Gecko/20110218 AlexaToolbar/alxf-2.0 Firefox/3.6.14",
"Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
"Mozilla/5.0(Macintosh;U;IntelMacOSX10_6_8;en-us)AppleWebKit/534.50(KHTML,likeGecko)Version/5.1Safari/534.50")
val ran = new Random().nextInt(agents.length)
agents(ran)
}
结尾唠叨两句
如果你对我的文章感兴趣，欢迎你点开我下一篇文章，后面我将手把手带你一起完成一个个小case，对了如果你也有好的想法，欢迎沟通交流
今天主要是带大家一起完成了知乎网站的爬取，练一练手，熟能生巧！