• urlgrabber: A highlevel crossprotocol urlgrabber


    urlgrabber: A high-level cross-protocol url-grabber

    urlgrabber

    urlgrabber is a pure python package that drastically simplifies
    the fetching of files. It is designed to be used in programs
    that need common (but not necessarily simple) url-fetching
    features. It is extremely simple to drop into an existing
    program and provides a clean interface to protocol-independant
    file-access. Best of all, urlgrabber takes care of all those
    pesky file-fetching details, and lets you focus on whatever it
    is that your program is written to do!

    urlgrabber came into existence as the part of yum that
    downloads rpms and header files, but it quickly became clear
    that this is a general problem that many applications must deal
    with.

    Features

    Using urlgrabber, data can be fetched in three basic ways:

    urlgrab(url) copy the file to the local filesystem
    urlopen(url) open the remote file and return a file object
    urlread(url) return the contents of the file as a string

    When using these functions (or methods), urlgrabber supports the
    following features:

    • identical behavior for http://, ftp://, and file:// urls
    • http keepalive - faster downloads of many files by using
      only a single connection
    • byte ranges - fetch only a portion of the file
    • reget - for a urlgrab, resume a partial download
    • progress meters - the ability to report download progress
      automatically, even when using urlopen!
    • throttling - restrict bandwidth usage
    • batched downloads using threads - download multiple files
      simultaneously (feature still in progress)
    • retries - automatically retry a download if it fails. The
      number of retries and failure types are configurable
    • authenticated server access for http and ftp
    • proxy support - support for authenticated http and ftp
      proxies
    • mirror groups - treat a list of mirrors as a single
      source, automatically switching mirrors if there is a
      failure
    • broad support - unix and windows, python 2.3 - 2.5 (it
      currently works for and is tested against 2.2 also, but that
      will be dropped if it becomes difficult)

    Not sure if urlgrabber is the tool for you? Check out our comparison of the major options.

    Documentation, Examples, and Help

    There are many sources of urlgrabber-related assistance and
    information

    • The urlgrabber package
      documentation
      , built from the __doc__ strings
      using pydoc
    • The examples page
    • Browsable urlgrabber
      git repo
    • The yum-devel
      mailing list. For now, urlgrabber is piggy-backing on this
      list. If it becomes necessary, we will get our own list.
      When posting to this list, please indicate that it is a
      urlgrabber-related post by beginning the subject with
      [UG].
  • 相关阅读:
    SQL语言基础
    selenium webdriver (python) 第二版
    selenium-webdriver(python) (十六) --unittest 框架
    从用户感知谈软件性能测试
    selenium webdriver (python) 第一版PDF
    selenium-webdriver(python) (十五) -- 鼠标事件
    selenium-webdriver(python) (十四) -- webdriver原理
    如何安装和配置 Rex-Ray?- 每天5分钟玩转 Docker 容器技术(74)
    如何实现跨 Docker 主机存储?- 每天5分钟玩转 Docker 容器技术(73)
    新书发布《每天5分钟玩转Docker容器技术》
  • 原文地址:https://www.cnblogs.com/lexus/p/2437325.html
Copyright © 2020-2023  润新知