• python中的concurrent.futures模块


    一 : 概述

      concurrent.futures模块提供了高度封装的异步调用接口

      ThreadPoolExecutor:线程池,提供异步调用

      ProcessPoolExecutor: 进程池,提供异步调用

      Both implement the same interface, which is defined by the abstract Executor class.

    二 : 基本方法

      submit(fn, *args, **kwargs) 异步提交任务

      map(func, *iterables, timeout=None, chunksize=1) 取代for循环submit的操作

      shutdown(wait=True) 相当于进程池的pool.close()+pool.join()操作, wait=True,等待池内所有任务执行完毕回收完资源后才继续 , wait=False,立即返回,并不会等待池内的任务执行完毕 , 但不管wait参数为何值,整个程序都会等到所有任务执行完毕 , submit和map必须在shutdown之前.

      result(timeout=None) 取得结果

      add_done_callback(fn) 添加回调函数

    #介绍
    The ProcessPoolExecutor class is an Executor subclass that uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock but also means that only picklable objects can be executed and returned.
    
    class concurrent.futures.ProcessPoolExecutor(max_workers=None, mp_context=None)
    An Executor subclass that executes calls asynchronously using a pool of at most max_workers processes. If max_workers is None or not given, it will default to the number of processors on the machine. If max_workers is lower or equal to 0, then a ValueError will be raised.
    
    
    #用法
    from concurrent.futures import ThreadPoolExecutor,ProcessPoolExecutor
    
    import os,time,random
    def task(n):
        print('%s is runing' %os.getpid())
        time.sleep(random.randint(1,3))
        return n**2
    
    if __name__ == '__main__':
    
        executor=ProcessPoolExecutor(max_workers=3)
    
        futures=[]
        for i in range(11):
            future=executor.submit(task,i)
            futures.append(future)
        executor.shutdown(True)
        print('+++>')
        for future in futures:
            print(future.result())
    
    ProcessPoolExecutor
    #介绍
    ThreadPoolExecutor is an Executor subclass that uses a pool of threads to execute calls asynchronously.
    class concurrent.futures.ThreadPoolExecutor(max_workers=None, thread_name_prefix='')
    An Executor subclass that uses a pool of at most max_workers threads to execute calls asynchronously.
    
    Changed in version 3.5: If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor.
    
    New in version 3.6: The thread_name_prefix argument was added to allow users to control the threading.Thread names for worker threads created by the pool for easier debugging.
    
    #用法
    与ProcessPoolExecutor相同
    
    ThreadPoolExecutor
    from concurrent.futures import ThreadPoolExecutor,ProcessPoolExecutor
    
    import os,time,random
    def task(n):
        print('%s is runing' %os.getpid())
        time.sleep(random.randint(1,3))
        return n**2
    
    if __name__ == '__main__':
    
        executor=ThreadPoolExecutor(max_workers=3)
    
        # for i in range(11):
        #     future=executor.submit(task,i)
    
        executor.map(task,range(1,12)) #map取代了for+submit
    
    map的用法
    from concurrent.futures import ThreadPoolExecutor,ProcessPoolExecutor
    from multiprocessing import Pool
    import requests
    import json
    import os
    
    def get_page(url):
        print('<进程%s> get %s' %(os.getpid(),url))
        respone=requests.get(url)
        if respone.status_code == 200:
            return {'url':url,'text':respone.text}
    
    def parse_page(res):
        res=res.result()
        print('<进程%s> parse %s' %(os.getpid(),res['url']))
        parse_res='url:<%s> size:[%s]
    ' %(res['url'],len(res['text']))
        with open('db.txt','a') as f:
            f.write(parse_res)
    
    
    if __name__ == '__main__':
        urls=[
            'https://www.baidu.com',
            'https://www.python.org',
            'https://www.openstack.org',
            'https://help.github.com/',
            'http://www.sina.com.cn/'
        ]
    
        # p=Pool(3)
        # for url in urls:
        #     p.apply_async(get_page,args=(url,),callback=pasrse_page)
        # p.close()
        # p.join()
    
        p=ProcessPoolExecutor(3)
        for url in urls:
            p.submit(get_page,url).add_done_callback(parse_page) #parse_page拿到的是一个future对象obj,需要用obj.result()拿到结果
    
    回调函数
  • 相关阅读:
    Netty源码分析之ByteBuf引用计数
    GitHub git push大文件失败(write error: Broken pipe)完美解决
    Windows10 Docker安装详细教程
    全面的Docker快速入门教程
    十本你不容错过的Docker入门到精通书籍推荐
    CentOS 8.4安装Docker
    postgres之一条sql查询总数及部分数据
    neo4j相关操作
    git上传大文件
    分布式文件系统fastdfs安装以及python调用
  • 原文地址:https://www.cnblogs.com/DoingBe/p/9545066.html
Copyright © 2020-2023  润新知