Python的并发并行[2] -> 队列[1] -> 使用队列进行任务控制

使用队列进行任务控制

---

1 FIFO与LIFO队列

FIFO(First In First Out)与LIFO(Last In First Out)分别是两种队列形式，在FIFO中，满足先入先出的队列方式，而LIFO则是后入先出的队列形式，利用这两种方式可以实现不同的队列功能。

from random import randint
from time import sleep, ctime
from queue import Queue, LifoQueue
from threading import Thread

COUNT = 0


class MyThread(Thread):
    """
    Bulid up a Module to make this subclass more general
    And get return value by add a function named 'getResult()'
    """
    def __init__(self, func, args, name=''):
        Thread.__init__(self)
        self.name = name
        self.func = func
        self.args = args

    def getResult(self):
        return self.res

    def run(self):
        print('Starting', self.name, 'at:', ctime())
        # Call function here and calculate the running time
        self.res = self.func(*self.args)
        print(self.name, 'finished at:', ctime())


class MyQueue():
    def __init__(self):
        self.funcs = [self.writer, self.reader]
        self.nfuncs = range(len(self.funcs))

    def writeQ(self, queue):
        global COUNT
        print('Producing object OBJ_%d for Q...' % COUNT, end=' ')
        queue.put('OBJ_%d' % COUNT, True)
        print('size now:', queue.qsize())
        COUNT += 1
    
    def readQ(self, queue):
        # If queue is empty, block here until queue available
        val = queue.get(True)
        print('Consumed object %s from Q... size now:' % val, queue.qsize())
    
    def writer(self, queue, loops):
        for i in range(loops):
            self.writeQ(queue)
            sleep(randint(1, 3))
    
    def reader(self, queue, loops):
        for i in range(loops):
            self.readQ(queue)
            sleep(randint(2, 5))
    
    def main(self):
        nloops = randint(2, 5)
        fifoQ = Queue(32)
        lifoQ = LifoQueue(32)
    
        # First In First Out mode for Queue
        print('-----Start FIFO Queue-----')
        threads = []
        for i in self.nfuncs:
            threads.append(MyThread(self.funcs[i], (fifoQ, nloops), self.funcs[i].__name__))
        for t in threads:
            t.start()
        for t in threads:
            t.join()
        # Last In First Out mode for LifoQueue
        print('-----Start LIFO Queue-----')
        threads = []
        for i in self.nfuncs:
            threads.append(MyThread(self.funcs[i], (lifoQ, nloops), self.funcs[i].__name__))
        for t in threads:
            t.start()
        for t in threads:
            t.join()
    
        print('All DONE')

if __name__ == '__main__':
     MyQueue().main()

第 1-27 行，首先对需要的模块进行导入，并定义一个全局变量的计数器，派生一个MyThread线程类，用于调用函数及其返回值（本例中MyThread可用于接受writer和reader函数，同时将Queue的实例作为参数传给这两个函数）。

第 30-79 行，定义一个队列类，用于进行队列一系列处理，其中writeQ与readQ会分别对队列执行put和get函数，在writeQ中利用全局变量设置每个加入队列的对象的名字。而writer和reader则会利用循环多次执行writeQ和readQ函数。最后定义一个main函数，用于生成队列，同时调用FIFO以及LIFO两种队列方式。

运行得到结果

-----Start FIFO Queue-----  
Starting writer at: Tue Aug  1 21:43:22 2017  
Producing object OBJ_0 for Q... size now: 1  
Starting reader at: Tue Aug  1 21:43:22 2017  
Consumed object OBJ_0 from Q... size now: 0  
Producing object OBJ_1 for Q... size now: 1  
Producing object OBJ_2 for Q... size now: 2  
Producing object OBJ_3 for Q... size now: 3  
Consumed object OBJ_1 from Q... size now: 2  
writer finished at: Tue Aug  1 21:43:26 2017  
Consumed object OBJ_2 from Q... size now: 1  
Consumed object OBJ_3 from Q... size now: 0  
reader finished at: Tue Aug  1 21:43:34 2017  
-----Start LIFO Queue-----  
Starting writer at: Tue Aug  1 21:43:34 2017  
Producing object OBJ_4 for Q... size now: 1  
Starting reader at: Tue Aug  1 21:43:34 2017  
Consumed object OBJ_4 from Q... size now: 0  
Producing object OBJ_5 for Q... size now: 1  
Producing object OBJ_6 for Q... size now: 2  
Producing object OBJ_7 for Q... size now: 3  
writer finished at: Tue Aug  1 21:43:38 2017  
Consumed object OBJ_7 from Q... size now: 2  
Consumed object OBJ_6 from Q... size now: 1  
Consumed object OBJ_5 from Q... size now: 0  
reader finished at: Tue Aug  1 21:43:53 2017  
All DONE  

从输出可以看出，FIFO满足先入先出，LIFO满足后入先出的队列形式。

2 join挂起与task_done信号

在queue模块中，Queue类提供了两个用于跟踪监测任务完成的函数，join和task_done，对于join函数来说，当Queue的类实例调用了join函数挂起时，join函数会阻塞等待，一直到join之前进入队列的所有任务全部标记为task_done后才会解除阻塞。

Note: 通过查看Queue的源码可以看出，在调用put函数时，会对类变量unfinished_tasks进行数值加1，而调用get函数时并不会将unfinished_tasks进行减1，只有调用task_done函数才会导致变量减1。而调用join函数时，join函数会对这个unfinished_tasks变量进行获取，也就是说，join函数会获取到在调用之前所有被put进队列里的任务中，还没有调用过task_done函数的任务数量，无论这个任务是否已经被get出列。

下面的例子中，以Queue_FIFO_LIFO.py中的MyQueue为基类，派生出一个新类，用于测试join函数与task_done函数。

from Queue_FIFO_LIFO import *

class NewQueue(MyQueue):
    def __init__(self):
        MyQueue.__init__(self)

    def writer(self, queue, loops):
        for i in range(loops):
            self.writeQ(queue)
            sleep(randint(1, 3))
        print('Producing join here, waiting consumer')
        queue.join()
    
    def reader(self, queue, loops):
        for i in range(loops):
            self.readQ(queue)
            sleep(randint(2, 5))
            print('OBJ_%d task done' % i)
            queue.task_done()

    def main(self):
        nloops = randint(2, 5)
        fifoQ = Queue(32)

        print('-----Start FIFO Queue-----')
        threads = []
        for i in self.nfuncs:
            threads.append(MyThread(self.funcs[i], (fifoQ, nloops), self.funcs[i].__name__))
        for t in threads:
            t.start()
        for t in threads:
            t.join()

        print('All DONE')

if __name__ == '__main__':
    NewQueue().main()

上面的代码，在导入模块后，调用MyQueue的初始化函数进行初始化设置。在新类NewQueue中，对原基类的writer和reader以及main方法进行了重载，加入了join函数和task_done函数，并在main函数中只采用FIFO队列进行试验。

运行得到结果

-----Start FIFO Queue-----  
Starting writer at: Wed Aug  2 09:06:40 2017  
Producing object OBJ_0 for Q... size now: 1  
Starting reader at: Wed Aug  2 09:06:40 2017  
Consumed object OBJ_0 from Q... size now: 0  
Producing object OBJ_1 for Q... size now: 1  
Producing object OBJ_2 for Q... size now: 2  
OBJ_0 task done  
Consumed object OBJ_1 from Q... size now: 1  
Producing object OBJ_3 for Q... size now: 2  
Producing object OBJ_4 for Q... size now: 3  
Producing join here, waiting consumer  
OBJ_1 task done  
Consumed object OBJ_2 from Q... size now: 2  
OBJ_2 task done  
Consumed object OBJ_3 from Q... size now: 1  
OBJ_3 task done  
Consumed object OBJ_4 from Q... size now: 0  
OBJ_4 task done  
reader finished at: Wed Aug  2 09:07:02 2017  
writer finished at: Wed Aug  2 09:07:02 2017  
All DONE  

通过得到的结果可以看出，当新类里的writer完成了自己的Producing任务后，会由join挂起，一直等待直到reader的Consuming全部完成且标记task_done之后，才会解除挂起，此时writer和reader将会一起结束退出。

相关阅读

---

1. 多线程的建立

2. queue 模块

参考链接 

---

 

《Python 核心编程 第3版》