python基础：multiprocessing的使用

不同于C++或Java的多线程，python中是使用多进程来解决多项任务并发以提高效率的问题，依靠的是充分使用多核CPU的资源。这里是介绍mulitiprocessing的官方文档：https://docs.python.org/2/library/multiprocessing.html

一、多进程并发效果演示

<span style="font-size:14px;">import multiprocessing
import time

def worker_1(ts):
    print "run worker_1"
    time.sleep(ts)
    print "end worker_1"

def worker_2(ts):
    print "run worker_2"
    time.sleep(ts)
    print "end worker_2"

def worker_3(ts):
    print "run worker_3"
    time.sleep(ts)
    print "end worker_3"

def worker_4(ts):
    print 'run worker_4'
    time.sleep(ts)
    print 'end worker_4'

def worker_5(ts):
    print 'run worker_5'
    time.sleep(ts)
    print 'end worker_5'

if __name__ == "__main__":
    proc1 = multiprocessing.Process(target = worker_1, args = (1,))
    proc2 = multiprocessing.Process(target = worker_2, args = (2,))
    proc3 = multiprocessing.Process(target = worker_3, args = (3,))
    proc4 = multiprocessing.Process(target = worker_4, args = (3,))
    proc5 = multiprocessing.Process(target = worker_5, args = (3,))

    proc1.start()
    proc2.start()
    proc3.start()
    proc4.start()
    proc5.start()

    print("The number of CPU is:" + str(multiprocessing.cpu_count()))
    for p in multiprocessing.active_children():
        print("child   p.name:" + p.name + "	p.id" + str(p.pid))
    print "main_process finished."</span>

运行结果：

分析：

通过上面的运行结果可以看到

（1）在主进程中start启动的5个进程彼此之间以及和主进程均存在并发关系，像上面worker_2在主进程的输出前输出，而且worker1、4、3、5分别无序输出‘run’就是并发最好的说明

（2）由于worker_1和worker_2分别sleep1秒和2秒，所以在主进程结束后依次结束，而worker_3、worker_4、worker_5都是sleep相同的3秒，最后它们三个进程无序输出（end4、end3、end5）更好的演示了并发效果

二、将进程写成class的范例

<span style="font-size:14px;">import multiprocessing
import time

class CounterProcess(multiprocessing.Process):
    def __init__(self, ts, arr):
        multiprocessing.Process.__init__(self)
        self.ts = ts
        self.arr = arr

    def run(self):
        time.sleep(self.ts)
        sum = 0
        for i in self.arr:
            sum += i
        print 'sum = ' + str(sum)

        c_time_cur_loc = time.localtime()
        counter_timestamp = '%04d%02d%02d_%02d%02d%02d' % (
            c_time_cur_loc.tm_year,
            c_time_cur_loc.tm_mon,
            c_time_cur_loc.tm_mday,
            c_time_cur_loc.tm_hour,
            c_time_cur_loc.tm_min,
            c_time_cur_loc.tm_sec
            )
        print 'counter_process finished at ' + str(counter_timestamp)

if __name__ == '__main__':
    arr = [1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
    ts = 2
    counter = CounterProcess(ts, arr)
    counter.start()

    for i in arr:
        print 'arr.member = ' + str(i)

    m_time_cur_loc = time.localtime()
    main_timestamp = '%04d%02d%02d_%02d%02d%02d' % (
        m_time_cur_loc.tm_year,
        m_time_cur_loc.tm_mon,
        m_time_cur_loc.tm_mday,
        m_time_cur_loc.tm_hour,
        m_time_cur_loc.tm_min,
        m_time_cur_loc.tm_sec
        )
    print 'main_process finished at ' + str(main_timestamp)</span>

运行结果：

分析：

这个范例是在主进程中一次输出数组中的斐波那契数列，然后由一个进程counter去计算该数列的累加和。

其中在进程初始化的时候设置了让该进程sleep两秒，然后在输出的结果中我们也可以看到主进程首先结束，然后在两秒后counter进程完成累加和的运算并且结束（累加和应该不到1ms，直接可以忽略，所以两个进程结束的时间差恰好就是我们预设的2秒）

三、daemon和join

（1）daemon：daemon的作用是控制主线程与其他线程的关系，默认情况下daemon=False，也就是当主进程关闭后，在主进程中start出来的进程会继续正常运行，而如果手动设置daemon=True，那么在主进程结束后，从主进程中start的所有其他进程进程也会立刻随着主进程的结束而结束。

<span style="font-size:14px;">import multiprocessing
import time

class CounterProcess(multiprocessing.Process):
    def __init__(self, ts, arr):
        multiprocessing.Process.__init__(self)
        self.ts = ts
        self.arr = arr

    def run(self):
        time.sleep(self.ts)
        sum = 0
        for i in self.arr:
            sum += i
        print 'sum = ' + str(sum)

        c_time_cur_loc = time.localtime(time.time())
        counter_timestamp = '%04d%02d%02d_%02d%02d%02d' % (
            c_time_cur_loc.tm_year,
            c_time_cur_loc.tm_mon,
            c_time_cur_loc.tm_mday,
            c_time_cur_loc.tm_hour,
            c_time_cur_loc.tm_min,
            c_time_cur_loc.tm_sec
            )
        print 'counter_process finished at ' + str(counter_timestamp)

if __name__ == '__main__':
    arr = [1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
    ts = 2
    counter = CounterProcess(ts, arr)
    counter.daemon = True
    counter.start()
    #counter.join()


    for i in arr:
        print 'arr.member = ' + str(i)

    m_time_cur_loc = time.localtime(time.time())
    main_timestamp = '%04d%02d%02d_%02d%02d%02d' % (
        m_time_cur_loc.tm_year,
        m_time_cur_loc.tm_mon,
        m_time_cur_loc.tm_mday,
        m_time_cur_loc.tm_hour,
        m_time_cur_loc.tm_min,
        m_time_cur_loc.tm_sec
        )
    print 'main_process finished at ' + str(main_timestamp)</span>

运行结果：

分析：

可以看到，设置了daemon=True后，并没有执行完正在sleep中的counter_process进程，而是随着main_process的结束而终止了。

（2）join：join的作用是阻塞当前进程，直到调用join的那个进程执行完它的运算，回到当前进程下继续执行当前进程。

<span style="font-size:14px;">import multiprocessing
import time

class CounterProcess(multiprocessing.Process):
    def __init__(self, ts, arr):
        multiprocessing.Process.__init__(self)
        self.ts = ts
        self.arr = arr

    def run(self):
        time.sleep(self.ts)
        sum = 0
        for i in self.arr:
            sum += i
        print 'sum = ' + str(sum)

        c_time_cur_loc = time.localtime(time.time())
        counter_timestamp = '%04d%02d%02d_%02d%02d%02d' % (
            c_time_cur_loc.tm_year,
            c_time_cur_loc.tm_mon,
            c_time_cur_loc.tm_mday,
            c_time_cur_loc.tm_hour,
            c_time_cur_loc.tm_min,
            c_time_cur_loc.tm_sec
            )
        print 'counter_process finished at ' + str(counter_timestamp)

if __name__ == '__main__':
    ms_time_cur_loc = time.localtime(time.time())
    main_s_timestamp = '%04d%02d%02d_%02d%02d%02d' % (
        ms_time_cur_loc.tm_year,
        ms_time_cur_loc.tm_mon,
        ms_time_cur_loc.tm_mday,
        ms_time_cur_loc.tm_hour,
        ms_time_cur_loc.tm_min,
        ms_time_cur_loc.tm_sec
        )
    print 'main_process started at ' + str(main_s_timestamp)

    arr = [1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
    ts = 2
    counter = CounterProcess(ts, arr)
    counter.daemon = True
    counter.start()
    counter.join()


    for i in arr:
        print 'arr.member = ' + str(i)

    me_time_cur_loc = time.localtime(time.time())
    main_e_timestamp = '%04d%02d%02d_%02d%02d%02d' % (
        me_time_cur_loc.tm_year,
        me_time_cur_loc.tm_mon,
        me_time_cur_loc.tm_mday,
        me_time_cur_loc.tm_hour,
        me_time_cur_loc.tm_min,
        me_time_cur_loc.tm_sec
        )
    print 'main_process finished at ' + str(main_e_timestamp)</span>

运行结果：

分析：

在本次执行的时候加入了主进程开始执行的时间，然后可以发现当在主进程中join了counter_process之后，就阻塞了当前正在运行的主进程，花了两秒时间完成了counter_process的运算，然后才继续进行main_process的运算，直到结束。

四、Lock

既然是并发，一定就会有lock来控制多进程访问共用资源的情况，python中锁有两种状态：被锁(locked)和没有被锁(unlocked)，拥有acquire( )和release( )两种方法，并且遵循以下的规则：

1、unlocked的锁 + acquire( ) = locked的锁

2、locked的锁 + acquire( ) = 调用acquire( )的进程将进入阻塞，直到其他进程调用release( )方法释放锁

3、unlocked的锁 + release( ) = 抛出RuntimeError异常

4、locked的锁 + release( ) = 将该锁的状态由locked转变成unlocked

感谢yoyzhou提供了一张很清晰的acquire( )和release( )的逻辑图，引用如下所示：

另外：锁（Lock）可以和"with"语句一起使用，锁可以作为上下文管理器（context manager）。

使用with的好处是：当程序执行到"with"语句的时候，acquire( )方法将被调用，当程序执行完"with"语句时，release( )方法将被调用。这样我们就不用显示的调用acqiure( )和release( )方法，而是由with语句根据上下文来管理锁的获取和释放。

<span style="font-size:14px;">import multiprocessing

file_strA = 'file_writerA is working'
file_strB = 'file_writerB is working'

def file_writerA(lock, file_path):
    print 'file_writerA process started already.'
    with lock:
        fs = open(file_path, 'a+')
        repeat_times = 1000000
        print 'file_writerA start to write.'
        while repeat_times >= 1:
            fs.write(file_strA + '
')
            repeat_times -= 1
        print 'file_writerA finished writing.'
        fs.close()


def file_writerB(lock, file_path):
    print 'file_writerB process started already.'
    lock.acquire()
    try:
        fs = open(file_path, 'a+')
        repeat_times = 1000000
        print 'file_writerB start to write.'
        while repeat_times >= 1:
            fs.write(file_strB + '
')
            repeat_times -= 1
        print 'file_writerB finished writing.'
        fs.close()
    finally:
        lock.release()

if __name__ == "__main__":
    mdr_lock = multiprocessing.Lock()
    file_path = "E:\file.txt"
    proc_writerA = multiprocessing.Process(target=file_writerA, args=(mdr_lock, file_path))
    proc_writerB = multiprocessing.Process(target=file_writerB, args=(mdr_lock, file_path))
    proc_writerA.start()
    proc_writerB.start()
    print "main_process is finished."</span>

运行结果：

分析：

上面程序中分别使用手动acquire( )/release( )和with两种写法控制两个进程去写相同的文件。

通过上面的运行结果可以看到，当file_writerA process启动以后，锁住了文件，此时file_writerB process也启动了，但是由于A没有完成写文件，所以B被阻塞，当A完成了写操作以后，B才开始继续执行自己的写文件命令。

另外需要强调一点，指定相同target的不同进程仍然是不同进程，也会被acquire阻塞住的，如下面实验所示。

<span style="font-size:14px;">import multiprocessing
import threading

file_strA = 'file_writerA is working'
file_strB = 'file_writerB is working'

def file_writerA(name, lock, file_path):
    print 'file_writer process ' + name + ' started already.'
    print 'inname = ' + multiprocessing.current_process().name
    with lock:
        fs = open(file_path, 'a+')
        repeat_times = 1000000
        print 'file_writer ' + name + ' start to write.'
        while repeat_times >= 1:
            fs.write(file_strA + '
')
            repeat_times -= 1
        print 'file_writer ' + name + ' finished writing.'
        fs.close()

def file_writerB(name, lock, file_path):
    print 'file_writer process ' + name + ' started already.'
    print 'inname = ' + multiprocessing.current_process().name
    lock.acquire()
    try:
        fs = open(file_path, 'a+')
        repeat_times = 1000000
        print 'file_writer ' + name + ' start to write.'
        while repeat_times >= 1:
            fs.write(file_strB + '
')
            repeat_times -= 1
        print 'file_writer ' + name + ' finished writing.'
        fs.close()
    finally:
        lock.release()

if __name__ == "__main__":
    mdr_lock = multiprocessing.Lock()
    file_path = "E:\file.txt"
    proc_writerA1 = multiprocessing.Process(target=file_writerA, args=('A1', mdr_lock, file_path,))
    proc_writerA2 = multiprocessing.Process(target=file_writerA, args=('A2', mdr_lock, file_path,))
    proc_writerB = multiprocessing.Process(target=file_writerB, args=('B', mdr_lock, file_path,))
    proc_writerA1.start()
    proc_writerA2.start()
    proc_writerB.start()
    print 'main_process is finished.'
</span>

运行结果：

分析：

上面的实验可以看到，proc_writerA1和pro_writerA2都是指定targer=file_writerA，但是从运行结果上我们看到A2被阻塞到A1和B都执行完毕才开始执行自己的写操作。

五、RLock

RLock是可重入锁(reetrant lock)，和Lock对比：

（1）相同之处：当某一个进程lock.acquire( )后，直到其释放前，其他所有acquire相同lock的进程将被阻塞（包括自身进程）。

（2）区别之处：是同一个进程能够不被阻塞的多次调用rlock.acquire( )，同样需要相等次数的release( )才能释放后，其他进程才可以结束acquire的阻塞。

参考文献：

http://www.cnblogs.com/kaituorensheng/p/4445418.html

http://www.cnblogs.com/lipijin/p/3709903.html

http://yoyzhou.github.io/blog/2013/02/28/python-threads-synchronization-locks/