Python之FTP多线程下载文件之多线程分块下载文件

Python中的ftplib模块用于对FTP的相关操作，常见的如下载，上传等。使用python从FTP下载较大的文件时，往往比较耗时，如何提高从FTP下载文件的速度呢？多线程粉墨登场，本文给大家分享我的多线程下载代码，需要用到的python主要模块包括：ftplib和threading。

首先讨论我们的下载思路，示意如下：

1. 将文件分块，比如我们打算采用20个线程去下载同一个文件，则需要将文件以二进制方式打开，平均分成20块，然后分别启用一个线程去下载一个块：

def setupThreads(self, filePath, localFilePath, threadNumber = 20):
    """
    set up the threads which will be used to download images
    list of threads will be returned if success, else
    None will be returned
    """
    try:
        temp = self.ftp.sendcmd('SIZE ' + filePath)
        remoteFileSize = int(string.split(temp)[1])
        blockSize = remoteFileSize / threadNumber
        rest = None
        threads = []
        for i in range(0, threadNumber - 1):
            beginPoint = blockSize * i
            subThread = threading.Thread(target = self.downloadFileMultiThreads, args = (i, filePath, localFilePath, beginPoint, blockSize, rest,))
            threads.append(subThread)
            
        assigned = blockSize * threadNumber
        unassigned = remoteFileSize - assigned
        lastBlockSize = blockSize + unassigned
        beginPoint = blockSize * (threadNumber - 1)
        subThread = threading.Thread(target = self.downloadFileMultiThreads, args = (threadNumber - 1, filePath, localFilePath, beginPoint, lastBlockSize, rest,))
        threads.append(subThread)
        return threads
    except Exception, diag:
        self.recordLog(str(diag), 'error')
        return None

其中的downloadFileMultiThreads函数如下：

def downloadFileMultiThreads(self, threadIndex, remoteFilePath, localFilePath, 
                                 beginPoint, blockSize, rest = None):
    """
    A sub thread used to download file
    """
    try:
        threadName = threading.currentThread().getName()
        # temp local file
        fp = open(localFilePath + '.part.' + str(threadIndex), 'wb')
        callback = fp.write
        
        # another connection to ftp server, change to path, and set binary mode
        myFtp = FTP(self.host, self.user, self.passwd)
        myFtp.cwd(os.path.dirname(remoteFilePath))
        myFtp.voidcmd('TYPE I')
        
        finishedSize = 0
        # where to begin downloading
        setBeginPoint = 'REST ' + str(beginPoint)
        myFtp.sendcmd(setBeginPoint)
        # begin to download
        beginToDownload = 'RETR ' + os.path.basename(remoteFilePath)
        connection = myFtp.transfercmd(beginToDownload, rest)
        readSize = self.fixBlockSize
        while 1:
            if blockSize > 0:
                remainedSize = blockSize - finishedSize
                if remainedSize > self.fixBlockSize:
                    readSize = self.fixBlockSize
                else:
                    readSize = remainedSize
            data = connection.recv(readSize)
            if not data:
                break
            finishedSize = finishedSize + len(data)
            # make sure the finished data no more than blockSize
            if finishedSize == blockSize:
                callback(data)
                break
            callback(data)
        connection.close()
        fp.close()
        myFtp.quit()
        return True
    except Exception, diag:
        return False

2. 等待下载完成之后我们需要对各个文件块进行合并，合并的过程见本系列之二：Python之FTP多线程下载文件之分块多线程文件合并

感谢大家的阅读，希望能够帮到大家！

Published by Windows Live Writer!