Python中提供了强大的Manage来做数据共享,其支持的类型非常多,包括value,Array,list,dict,queue,lock等
例如:
import multiprocessing def worker(d,l): l += xrange(1,5) for i in xrange(11,15): key = "key{0}".format(i) val = "val{0}".format(i) d[key] = val # print d # print l if __name__ == "__main__": manager = multiprocessing.Manager() d = manager.dict() l = manager.list() p = multiprocessing.Process(target=worker,args=(d,l)) p.start() p.join() print d print l
运行结果:
{'key14': 'val14', 'key13': 'val13', 'key12': 'val12', 'key11': 'val11'} [1, 2, 3, 4]
进程池
Pool可以提供指定数量的进程,供用户调用,当有新的请求提交到pool中时,如果池还没有满,那么就会创建一个新的进程用来执行该请求;但如果池中的进程数已经达到规定最大值,那么该请求就会等待,直到池中有进程结束,才会创建新的进程。
阻塞和非阻塞的区别:
Pool.apply_async 非阻塞,定义的进程池进程最大数可以同时执行。
Pool.apply 一个进程结束,释放回进程池,下一个进程才可以开始
import multiprocessing import time def worker(msg): print "##### start {0} #####".format(msg) time.sleep(1) print "##### end {0} #####".format(msg) if __name__ == "__main__": print "main start" pool = multiprocessing.Pool(processes=3) for i in xrange(1,10): msg = "hello {0}".format(i) pool.apply_async(func=worker,args=(msg,)) # pool.apply(func=worker,args=(msg,)) pool.close() pool.join() #join一直一定要close ,否则报错 print "main end"
运行结果:
main start ##### start hello 1 ##### ##### start hello 2 ##### ##### start hello 3 ##### ##### end hello 1 ##### ##### start hello 4 ##### ##### end hello 2 ##### ##### start hello 5 ##### ##### end hello 3 ##### ##### start hello 6 ##### ##### end hello 4 ##### ##### start hello 7 ##### ##### end hello 5 ##### ##### start hello 8 ##### ##### end hello 6 ##### ##### start hello 9 ##### ##### end hello 7 ##### ##### end hello 8 ##### ##### end hello 9 ##### ##### end hello 9 #####