• 使用Python3 协程语法 async await 来实现协程异步http请求


    都知道Python的多任务有些尴尬,多进程可以用多核,但是消耗大,线程吧,无能用多核,是全局解释器锁来回切,所以通常都比较青睐协程了,但是协程是基于生成器的,不使用第三方库的开发成本学习成本就上去了,目前用的多的就是Gevent,基于Greenlet,使用类似于线程,不过在Python3.5以上版本Python提供了协程语法,可以更方便的使用,但是新的语法和一般的Python语法又有些不同,使用新的关键字,async,await

    下面以一个例子来实现协程异步操作http请求

     1 import asyncio
     2 import traceback
     3 import aiohttp
     4 
     5 
     6 Normal = "http://github.com/"
     7 
     8 async def get_url(url):
     9     client = aiohttp.ClientSession()
    10     try:
    11         resp = await client.get(url)
    12         await client.close()
    13         return resp
    14     except:
    15         await client.close()
    16         return traceback.format_exc()
    17 
    18 async def post_url(url, data:dict=None):
    19     async with aiohttp.ClientSession() as client:
    20         try:
    21             async with client.post(url, data=data) as resp:
    22                 return resp
    23         except:
    24             return traceback.format_exc()
    25 
    26 async def ask_url(url=Normal):
    27     print("%s===Will Await..."%url)
    28     response = await get_url(url)
    29     print("%s===Stop . . ."%url)
    30     print(response)
    31 
    32 
    33 if __name__ == '__main__':
    34     urls = [Normal, "http://baidu.com", "http://zhihu.com"]
    35     loop = asyncio.get_event_loop()
    36     tasks = [asyncio.ensure_future(ask_url(x)) for x in urls]
    37     loop.run_until_complete(asyncio.wait(tasks))
    38     loop.close()

    执行结果如下:

    http://github.com/===Will Await...
    http://baidu.com===Will Await...
    http://zhihu.com===Will Await...
    http://baidu.com===Stop . . .
    <ClientResponse(http://baidu.com) [200 OK]>
    <CIMultiDictProxy('Date': 'Sat, 19 Jun 2021 14:39:29 GMT', 'Server': 'Apache', 'Last-Modified': 'Tue, 12 Jan 2010 13:48:00 GMT', 'Etag': '"51-47cf7e6ee8400"', 'Accept-Ranges': 'bytes', 'Content-Length': '81', 'Cache-Control': 'max-age=86400', 'Expires': 'Sun, 20 Jun 2021 14:39:29 GMT', 'Connection': 'Keep-Alive', 'Content-Type': 'text/html')>
    
    http://zhihu.com===Stop . . .
    <ClientResponse(https://www.zhihu.com/signin?next=/) [200 OK]>
    <CIMultiDictProxy('Server': 'CLOUD ELB 1.0.0', 'Date': 'Sat, 19 Jun 2021 14:39:29 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Vary': 'Accept-Encoding', 'content-security-policy': "default-src * blob:; img-src * data: blob: resource: t.captcha.qq.com cstaticdun.126.net necaptcha.nosdn.127.net; connect-src * wss: blob: resource:; frame-src 'self' *.zhihu.com mailto: tel: weixin: *.vzuu.com mo.m.taobao.com getpocket.com note.youdao.com safari-extension://com.evernote.safari.clipper-Q79WDW8YH9 zhihujs: captcha.guard.qcloud.com pos.baidu.com dup.baidustatic.com openapi.baidu.com wappass.baidu.com passport.baidu.com *.cme.qcloud.com vs-cdn.tencent-cloud.com t.captcha.qq.com c.dun.163.com; script-src 'self' blob: *.zhihu.com g.alicdn.com qzonestyle.gtimg.cn res.wx.qq.com open.mobile.qq.com 'unsafe-eval' unpkg.zhimg.com unicom.zhimg.com resource: captcha.gtimg.com captcha.guard.qcloud.com pagead2.googlesyndication.com cpro.baidustatic.com pos.baidu.com dup.baidustatic.com i.hao61.net 'nonce-f5ee3225-450b-407f-898c-12f7faa96fb7' hm.baidu.com zz.bdstatic.com b.bdstatic.com imgcache.qq.com vs-cdn.tencent-cloud.com gw.alipayobjects.com ssl.captcha.qq.com t.captcha.qq.com cstaticdun.126.net c.dun.163.com ac.dun.163.com/ acstatic-dun.126.net; style-src 'self' 'unsafe-inline' *.zhihu.com unicom.zhimg.com resource: captcha.gtimg.com ssl.captcha.qq.com t.captcha.qq.com cstaticdun.126.net c.dun.163.com ac.dun.163.com/ acstatic-dun.126.net", 'x-content-security-policy': "default-src * blob:; img-src * data: blob: resource: t.captcha.qq.com cstaticdun.126.net necaptcha.nosdn.127.net; connect-src * wss: blob: resource:; frame-src 'self' *.zhihu.com mailto: tel: weixin: *.vzuu.com mo.m.taobao.com getpocket.com note.youdao.com safari-extension://com.evernote.safari.clipper-Q79WDW8YH9 zhihujs: captcha.guard.qcloud.com pos.baidu.com dup.baidustatic.com openapi.baidu.com wappass.baidu.com passport.baidu.com *.cme.qcloud.com vs-cdn.tencent-cloud.com t.captcha.qq.com c.dun.163.com; script-src 'self' blob: *.zhihu.com g.alicdn.com qzonestyle.gtimg.cn res.wx.qq.com open.mobile.qq.com 'unsafe-eval' unpkg.zhimg.com unicom.zhimg.com resource: captcha.gtimg.com captcha.guard.qcloud.com pagead2.googlesyndication.com cpro.baidustatic.com pos.baidu.com dup.baidustatic.com i.hao61.net 'nonce-f5ee3225-450b-407f-898c-12f7faa96fb7' hm.baidu.com zz.bdstatic.com b.bdstatic.com imgcache.qq.com vs-cdn.tencent-cloud.com gw.alipayobjects.com ssl.captcha.qq.com t.captcha.qq.com cstaticdun.126.net c.dun.163.com ac.dun.163.com/ acstatic-dun.126.net; style-src 'self' 'unsafe-inline' *.zhihu.com unicom.zhimg.com resource: captcha.gtimg.com ssl.captcha.qq.com t.captcha.qq.com cstaticdun.126.net c.dun.163.com ac.dun.163.com/ acstatic-dun.126.net", 'x-webkit-csp': "default-src * blob:; img-src * data: blob: resource: t.captcha.qq.com cstaticdun.126.net necaptcha.nosdn.127.net; connect-src * wss: blob: resource:; frame-src 'self' *.zhihu.com mailto: tel: weixin: *.vzuu.com mo.m.taobao.com getpocket.com note.youdao.com safari-extension://com.evernote.safari.clipper-Q79WDW8YH9 zhihujs: captcha.guard.qcloud.com pos.baidu.com dup.baidustatic.com openapi.baidu.com wappass.baidu.com passport.baidu.com *.cme.qcloud.com vs-cdn.tencent-cloud.com t.captcha.qq.com c.dun.163.com; script-src 'self' blob: *.zhihu.com g.alicdn.com qzonestyle.gtimg.cn res.wx.qq.com open.mobile.qq.com 'unsafe-eval' unpkg.zhimg.com unicom.zhimg.com resource: captcha.gtimg.com captcha.guard.qcloud.com pagead2.googlesyndication.com cpro.baidustatic.com pos.baidu.com dup.baidustatic.com i.hao61.net 'nonce-f5ee3225-450b-407f-898c-12f7faa96fb7' hm.baidu.com zz.bdstatic.com b.bdstatic.com imgcache.qq.com vs-cdn.tencent-cloud.com gw.alipayobjects.com ssl.captcha.qq.com t.captcha.qq.com cstaticdun.126.net c.dun.163.com ac.dun.163.com/ acstatic-dun.126.net; style-src 'self' 'unsafe-inline' *.zhihu.com unicom.zhimg.com resource: captcha.gtimg.com ssl.captcha.qq.com t.captcha.qq.com cstaticdun.126.net c.dun.163.com ac.dun.163.com/ acstatic-dun.126.net", 'x-frame-options': 'SAMEORIGIN', 'strict-transport-security': 'max-age=15552000; includeSubDomains', 'surrogate-control': 'no-store', 'Pragma': 'no-cache', 'Expires': '0', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'X-Backend-Response': '0.027', 'Referrer-Policy': 'no-referrer-when-downgrade', 'X-SecNG-Response': '0.030999898910522', 'x-lb-timing': '0.033', 'x-idc-id': '2', 'Set-Cookie': 'KLBRSID=d017ffedd50a8c265f0e648afe355952|1624113569|1624113569; Path=/', 'Content-Encoding': 'gzip', 'Cache-Control': 'private, must-revalidate, no-cache, no-store, max-age=0', 'Transfer-Encoding': 'chunked', 'X-NWS-LOG-UUID': '4935272354838850546', 'Connection': 'keep-alive', 'X-Cache-Lookup': 'Cache Miss', 'x-edge-timing': '0.063', 'x-cdn-provider': 'tencent')>
    
    http://github.com/===Stop . . .
    Traceback (most recent call last):
      File "D:/AppData/Python/ForAsync/main.py", line 12, in get_url
        resp = await client.get(url)
      File "C:UsersHaitonAppDataLocalProgramsPythonPython38libsite-packagesaiohttpclient.py", line 544, in _request
        await resp.start(conn)
      File "C:UsersHaitonAppDataLocalProgramsPythonPython38libsite-packagesaiohttpclient_reqrep.py", line 890, in start
        message, payload = await self._protocol.read()  # type: ignore
      File "C:UsersHaitonAppDataLocalProgramsPythonPython38libsite-packagesaiohttpstreams.py", line 604, in read
        await self._waiter
    aiohttp.client_exceptions.ClientOSError: [WinError 121] 信号灯超时时间已到

    可以看到,同一个线程中,并么有挨个执行,而是遇到相应差的和需要IO读写的自动切换了,其中GitHub最先遇到,但是最后执行结束(还是因为超时报错),所以新语法解决协程异步IO是有效的。

    上面的POST和GET通过不同的方式实现,是为了展示两种方式,因为aiohttp的 ClientSession 是一个IO的上下文,像文件一样,需要关闭IO,可以像get_url中一样,手动close,也可以像post_url一样,通过with来自动关闭。

  • 相关阅读:
    jQuery中$(function(){})与(function($){})(jQuery)、$(document).ready(function(){})等的区别详细讲解
    jQuery的三种$()方式
    基于DDD的.NET开发框架
    你得学会并且学得会的Socket编程基础知识
    C#委托及事件
    C#学习笔记:泛型委托Action<T>和Fun<TResult>
    JS 变量或参数是否有值的判断
    [转]剖析ASP.Net MVC Application
    python爬虫beautifulsoup4系列2【转载】
    python爬虫beautifulsoup4系列1【转载】
  • 原文地址:https://www.cnblogs.com/haiton/p/14905253.html
Copyright © 2020-2023  润新知