• Python3并发写文件


    使用python2在进行并发写的时候,发现文件会乱掉,就是某一行中间会插入其他行的内容。
    但是在使用python3进行并发写的时候,无论是多进程,还是多线程,都没有出现这个问题,难道是python3的特性吗?

    import time
    import os
    import multiprocessing
    from multiprocessing.dummy import Pool as ThreadPool
    
    
    def write(val, file):
        w = open(file, "a")
        for i in range(100):
            w.write("%s
    " % val)
            time.sleep(0.001)
    
    def thread_write(file):
        res, pools = [], ThreadPool(10)
        for i in range(10):
            val = str(i) * 1000
            res.append(pools.apply_async(func=write, args=(val, file, )))
    
        while res:
            for ret in res:
                if ret.ready():
                    res.remove(ret)
            time.sleep(0.01)
    
    def mutil_write(file):
        pools = multiprocessing.Pool(processes=10)
        res = []
        for i in range(100):
            res.append(pools.apply_async(thread_write, args=(file, )))
    
        while res:
            for ret in res:
                if ret.ready():
                    res.remove(ret)
            time.sleep(0.01)
    
    if __name__ == '__main__':
        file = "./write_test"
        mutil_write(file)
    
        with open(file) as fb:
            lines = 0
            line_len = []
            for line in fb:
                lines += 1
                line = line.strip()
                line_len.append(len(line))
                if len(line) != 1000:
                    raise(Exception("error line: %s, len: %d" % (line, len(line))))
    
            print("lines:%d, max len:%d, min:%d, avg:%.2f" % (lines, max(line_len), min(line_len), sum(line_len)/len(line_len)))
        os.remove(file)
    

    上面代码,多进程并发写结束后,校验每一行的长度是否是设置好的长度。用python3反复运行,均通过测试没有异常。

    $ python3 --version
    Python 3.7.4
    
    $ python3 t.py
    lines:10000, max len:1000, min:1000, avg:1000.00
    

    如果用python2,则会出现异常:

    $ python2 --version
    Python 2.7.15
    
    $ python2 t.py
    Traceback (most recent call last):
      File "t.py", line 49, in <module>
        raise(Exception("error line: %s, len: %d" % (line, len(line))))
    Exception: error line: 333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333330000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, len: 1092
    
  • 相关阅读:
    SQL面试积累
    The server does not support version 3.0 of the J2EE Web module specification
    AspectJ报错:error at ::0 can't find referenced pointcut XXX
    maven教程
    sqlMapConfig.xml配置文件详解
    解决eclipse+MAVEN提示One or more constraints have not been satisfied.的问题
    maven常见问题
    Dmaven.multiModuleProjectDirectory system propery is not set.
    maven最齐全配置pom.xml
    serialVersionUID的作用
  • 原文地址:https://www.cnblogs.com/yzhch/p/11896859.html
Copyright © 2020-2023  润新知