urlgrabber: A high-level cross-protocol url-grabber
urlgrabber
urlgrabber is a pure python package that drastically simplifies
the fetching of files. It is designed to be used in programs
that need common (but not necessarily simple) url-fetching
features. It is extremely simple to drop into an existing
program and provides a clean interface to protocol-independant
file-access. Best of all, urlgrabber takes care of all those
pesky file-fetching details, and lets you focus on whatever it
is that your program is written to do!urlgrabber came into existence as the part of yum that
downloads rpms and header files, but it quickly became clear
that this is a general problem that many applications must deal
with.Features
Using urlgrabber, data can be fetched in three basic ways:
urlgrab(url) copy the file to the local filesystem urlopen(url) open the remote file and return a file object urlread(url) return the contents of the file as a string When using these functions (or methods), urlgrabber supports the
following features:
- identical behavior for http://, ftp://, and file:// urls
- http keepalive - faster downloads of many files by using
only a single connection- byte ranges - fetch only a portion of the file
- reget - for a urlgrab, resume a partial download
- progress meters - the ability to report download progress
automatically, even when using urlopen!- throttling - restrict bandwidth usage
- batched downloads using threads - download multiple files
simultaneously (feature still in progress)- retries - automatically retry a download if it fails. The
number of retries and failure types are configurable- authenticated server access for http and ftp
- proxy support - support for authenticated http and ftp
proxies- mirror groups - treat a list of mirrors as a single
source, automatically switching mirrors if there is a
failure- broad support - unix and windows, python 2.3 - 2.5 (it
currently works for and is tested against 2.2 also, but that
will be dropped if it becomes difficult)Not sure if urlgrabber is the tool for you? Check out our comparison of the major options.
Documentation, Examples, and Help
There are many sources of urlgrabber-related assistance and
information
- The urlgrabber package
documentation, built from the __doc__ strings
using pydoc- The examples page
- Browsable urlgrabber
git repo- The yum-devel
mailing list. For now, urlgrabber is piggy-backing on this
list. If it becomes necessary, we will get our own list.
When posting to this list, please indicate that it is a
urlgrabber-related post by beginning the subject with
[UG].