Python爬虫Scrapy框架(3) -- 反爬虫

爬取代理

Python3中urllib详细使用方法(header,代理,超时,认证,异常处理)，详见https://www.cnblogs.com/ifso/p/4707135.html

验证代理

import urllib.request
import re
import threading


class TestProxy(object):
    def __init__(self):
        self.sFile = r'proxy.txt'
        self.dFile = r'alive.txt'
        self.URL = r'http://www.baidu.com/'
        self.threads = 10
        self.timeout = 3
        self.regex = re.compile(r'baidu.com')
        self.aliveList = []

        self.run()

    def run(self):
        with open(self.sFile, 'r') as fp:
            lines = fp.readlines()
            line = lines.pop()
            while lines:
                for i in range(self.threads):
                    t = threading.Thread(target=self.linkWithProxy, args=(line,))
                    t.start()
                    if lines:
                        line = lines.pop()
                    else:
                        continue

        with open(self.dFile, 'w') as fp:
            for i in range(len(self.aliveList)):
                fp.write(self.aliveList[i])



    def linkWithProxy(self, line):
                lineList=line.split('	')
                protocol=lineList[2].lower()
                server=protocol+r'://'+lineList[0]+':'+lineList[1]
                opener=urllib.request.build_opener(urllib.request.ProxyHandler({protocol:server}))
                urllib.request.install_opener(opener)
                try:
                        response=urllib.request.urlopen(self.URL,timeout=self.timeout)
                except:
                        print('%s connect faild' %server)
                        return
                else:
                        try:
                                strli=response.read()51                         except:
                                print('%s connect failed' %server)
                                return
                        if self.regex.search(strli):
                                print('%s connect success ..........' %server)
                                self.aliveList.append(line)

if __name__ == '__main__':
    TP = TestProxy()

第50行报错，TypeError: cannot use a string pattern on a bytes-like object 

改成 strli=response.read().decode('utf-8') 

反爬虫

1、robots协议：当爬虫访问一个站点时，它会检查该目录下是否存在robot.txt，如果存在，按照文件的内容确定访问范围

      解决方法：A. 伪装浏览器   B.设置setting文件， ROBOTSTXT_OBEY = False  

2、IP流量异常：网站发现IP流程异常增多时就会封IP

     解决方法：A. 增大爬取间隔时间，设置随机时间  B.更改IP