python 在内存中处理tar.bz2文件

如果tar.bz2文件是通过网络进行下载，那么可以直接在内存进行解压后读取文件内容，不用将文件缓存到本地然后解压再进行读取，可以节省IO。

处理经过gzip压缩的tar文件的方法见：https://stackoverflow.com/questions/15352668/download-and-decompress-gzipped-file-in-memory

处理bz2压缩的tar文件的方法：https://stackoverflow.com/questions/46291529/how-to-decompress-tar-bz2-in-memory-with-python

import requests
import tarfile
from io import BytesIO

url = "www.google.com"   # the url you get tar.bz2 file from, need to change according to your application.
filename = "res_test.csv"  # the filename in your tar.bz2 file.


def decompress_tar_bz2_from_net(url, filename):
    """
    decompress the tar.bz2 format file in memory, instead of buffer it on disk
    and then decompress.
    :param url:
    :param filename:
    :return:
    """
    fileobj = BytesIO(requests.get(url).content)
    contents = tarfile.open(fileobj=fileobj).extractfile(filename).read()
    return contents

更多使用方法见：https://github.com/buxizhizhoum/tool_scripts/blob/master/app/bin/decompress_in_memory.py