An abstract class represents an interface.
Interfaces are the subject of this chapter: from the dynamic protocols that are the hallmark of ducking typing to abstract base classes (ABCs) that make interfaces explicit and verify implementations for conformance.
If you have a Java, C#, or similar background , the novelty here is in the informal protocols of duck typing. But for the long-time Pythonista or Rubyist, that is the "normal" way of thinking about interfaces, and the news is the formality and type-checking of ABCs.
We'll start the chapter by reviewing how the Python community traditionally understood interfaces as somewhat loose--in the sense that a partially implemented interface is often acceptable. We'll make that clear through a couple examples that highlight the dynamic nature of duck typing.
The rest of the chapter will be devoted to ABCs, starting with their common use as superclass when you need to implement an interface. We'll then see when an ABC checks concrete subclass for conformance to the interface it defines, and how a registration mechanism lets developers declare that a class implements an interface without subclassing. Finally, we'll see how an ABC can be programmed to automatically "recognize" arbitrary classes that conform to its interface--without subclassiing or explicit registration.
Starting writing your own ABCs is not encouraged. The risk of over-engineering with ABCs is very high.
1. Interfaces and Protocols in Python Culture
Protocols are defined as the informal interfaces that make polymorphism work in languages with dynamic typing like Python.
How do interfaces work in a dynamic-types languages ? Firstly, the basics: even without an interface keyword in the language, and regardless of ABCs, every class has an interface: the set public attributes(methods or data attributes) implemented or inherited by the class. This includes special methods, like __getitem__ or __add__ .
By definition, protected and private attributes are not part of an interface, even if "protected" is merely a naming convention and private attributes are easily accessed. It's bad form to violate these conventions.
A useful complementary definition of interface is: the subset of an object's public methods that enable it to play a specific role in the system. That's what is implied when the Python codumentation mentions "a file-like object" or "an iterable", without specifying a class. An interface seen as a set of methods to fulfill a role is what Smalltalkers called a protocol, and the term spread to other dynamic language communities. Protocols are independent of inheritance. A class may implement several protocols, enabling its instances to fulfill several roles.
Protocols are interfaces, but because they are informal--defined only by documentation and conventions--protocols cannot be enfored like formal interfaces can(we'll see how ABCs enforce interface conformance). A protocol may be partially implemented in a particular class, and that's OK. Sometimes all a specific API requires from "a file-like object" is that it has a .read() method that returns bytes. The remaining file methods may or may not be relevant in the context.
"X-like object", "X protocol", and "X interface" are synonyms in the minds f Pythonistas.
2. Python Digs Sequences
The philosophy of the Python data model is to cooperate with essential protocols as much as possible. When it comes to sequences, Python tries hard to work with even the simplest implementations.
Now take a look at the Foo class in Example 11-3. It does not inherit from abc.Sequence, and it only implements one method of the sequence protocol: __getitem__ (__len__ is missing).
Example 11-3. Partial sequence protocol implementation with __getitem__ : enough for item access, iteration, and the in operator.
>>> class Foo: ... def __getitem__(self, pos): ... return range(0,30,10)[pos] ... >>> f = Foo() >>> f[1] 10 >>> for i in f: print(i) ... 0 10 20 >>> 20 in f True >>> 15 in f False """ There is no method __iter__ yet Foo instances are iterable because--as a fallback--when Python sees a __getitem__ method, it tries to iterate over the object by calling that method with integer indexes starting with 0. Because Python is smart enough to iterate over Foo instances, it can also make the in operator work even if Foo has no __contains__ method: it does a full scan to check if an item is present. In summary, given the importance of the sequence protocol, in the absence of __iter__ and __contains__ Python still manages to make iteration and the in operator work by invoking __getitem__ . """
A good part of the demos in Chapter 1 work because of the special treatment Python gives to anything vaguely resembling a sequence. Iteration in Python represents an extreme form of duck typing: the interpreter tries two different methods to iterate over objects.
3. Mokey-Patching to Implement a Protocol at Runtime
The FrenchDeck class from Example 1-1 has a major flaw: it cannot be shffled. Years ago when I first wrote the FrenchDeck example I did implement a shuffle method. Later I had a Pythonic insight: if a FrenchDeck acts like a sequence, then it doesn't need its own shuffle method because there is already random.shuffle, documented as "Shuffle the sequence x in place".
However, if we try to shuffle a FrenchDeck instance, we get an exception, as in Example 11-5.
Codes of Example 1-1.
import collections Card = collections.namedtuple("Card", ["rank", "suit"]) class FrenchDeck(object): ranks = [str(n) for n in range(2, 11)] + list("JQKA") suits = "spades diamonds clubs hearts".split() def __init__(self): self._cards = [Card(rank, suit) for suit in self.suits for rank in self.ranks] def __len__(self): return len(self._cards) def __getitem__(self, position): return self._cards[position]
Example 11-5. random.shuffle cannot handle FrenchDeck
>>> from random import shuffle >>> from ex1_1_frenchdeck import FrenchDeck >>> >>> deck = FrenchDeck() >>> >>> shuffle(deck) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/random.py", line 274, in shuffle x[i], x[j] = x[j], x[i] TypeError: 'FrenchDeck' object does not support item assignment >>>
The error messages is quite clear: "'FrenchDeck' object does not support item assignment." The problem is that shuffle operates by swapping items inside the collection, and FrenchDeck only implements this immutable sequence protocol. Mutable sequences must also provide a __setitem__ method.
Because Python is dynamic, we can fix this at runtime, even at the interactive console. Example 11-6 shows how to do it.
Example 11-6. Monkey patching FrenchDeck to make it mutable and compatible with random.shuffle (continuing from Example 11-5)
>>> def set_card(deck, position, card): # the arguments deck, position, card can be replaced with self, key, value. ... deck._cards[position] = card ... >>> FrenchDeck.__setitem__ = set_card # Assign the function of set_card to an attribute named __setitem__ in the FrenchDeck class. >>> shuffle(deck) # deck can now be sorted because FrenchDeck now implements the necessary method of the mutable sequence protocol -- __setitem__ . >>> deck[:5] [Card(rank='10', suit='spades'), Card(rank='A', suit='hearts'), Card(rank='K', suit='spades'), Card(rank='9', suit='diamonds'), Card(rank='6', suit='hearts')] >>> """ The trick in this example is that set_card knows that the the deck object has an attribute named _cards, and _cards must be mutable sequence. The set_card function is then attached to the FrenchDeck class as the __setitem__ special method. This is an example of monkey patching: changing a class or module at runtime, without touching the source code. Monkey patching is powerful, but the code that does the actual patching is very tightly coupled with the program to be patched, often handling private and undocumented parts. This example also highlights that protocols are dynamic: random.shuffle doesn't care what type of argument it gets, it only needs the object to implement part of the mutable sequence protocol. It doesn't even matter if the object was "born" with the necessary methods or if they were somehow acquired later. The theme of this chapter so far has been "duck typing": operating with objects regardless of their types, as long as they implement certain protocols. """
4. Alex Martelli's Waterfowl
After reviewing the usual protocol-style interfaces of Python, we move to ABCs.
Excerpt from "Waterfowl and ABCs" by Alex Martelli
ducking typing: ignoring an object's actual type, focusing instead on ensuring that the object implements the method names, signatures, and semantics required for its intended use. In Python, this mostly boils down to avoiding the use of isinstance to check the the object's type(not to mention the even worse approach of checking, for example, whether type(foo) is bar -- which is rightly anathema as it inhibits even the simplest forms of inheritance). (type 连最简单的继承都判断不出来,如下：) """ In [1]: class A: ...: pass ...: In [2]: class B(A): ...: ... ...: In [3]: b = B() In [4]: type(b) Out[4]: __main__.B In [5]: type(b) is B Out[5]: True In [6]: type(b) is A Out[6]: False In [7]: isinstance(b,A) Out[7]: True """ I am recommending supplementing (not entirely replacing -- in certain contexts it shall still serve) good old duck typing with ... goose typing. What goose typing means is: isinstance(obj, cls) is now just fine... as long as cls is an abstract base class -- in other words, cls's metaclass is abc.ABCMeta. Python's ABCs add one major practical advantage: the register class method, which lets end-user code "declare" that a certain class becomes a "virtual" subclass of an ABC (for this purpose the registered class must meet the ABC's method name and signature requirements, and more importantly the underlying semantics contract -- but it need not have been developed with any awareness of the ABC, and in particular need not inherit from it!). Sometimes you don't even need to register a class fro an ABC to recognize it as a subclass! That's the case for the ABCs whose essence boils down to a few special methods. For example: """ >>> class Struggle: ... def __len__(self): return 23 ... >>> from collections import abc >>> isinstance(Struggle(), abc.Sized) True >>> class Ctner: ... def __contains__(self, item): ... return True ... >>> isinstance(Ctner(), abc.Container) True >>> 'a' in 'abc' True >>> 'a' in Ctner() True """ abc.Sized recognizes Struggle as "a subclass", with no need for registration, as implementing the special method named __len__ is all it takes(it's supposed to be implemented with the proper syntax -- callable without arguments -- and semantics -- returning a non-negative integer denoting an object's "length") Whenever you're implementing a class embodying any of the concepts represented in the ABCs in numbers, collections.abc, or other framework you may be using, be sure(if needed) to subclass it from, or register it into, the corresponding ABC. At the start of your programs using some library or framework defining classes which have omitted to do that,perform the registration yourself; then,when you must check for (mostly typically) an argument being, e.g. "a sequence," check whether: isinstance(the_arg, collections.abc.Sequence)
Besides coining the "goose typing", Alex makes the point that inheriting from an ABC is more than implementing the required methods: it's also a clear declaration of intent by developer. That intent can also be made explicit through registering a virtual subclass. In addition, the use of isinstance and issubclass becomes more acceptable to test against ABCs. In the past, these functions worked against duck typing, but with ABCs they become more flexible. After all, if a component does not implement an ABC by subclassing, it can always be registered after the fact so it passes those explicit type checks.
However, even with ABCs, you should beware that excessive use of isinstance checks may be a code smell -- a symptom of bad OO design. It's usually not OK to have a chain of if/elif/elif with isinstance checks performing different actions depend on the type of an object: you should be using polymorphism for that -- i.e., designing your classes so that the interpreter dispatches calls to the proper methods, instead of you hardcoding the dispatch logic in if/elif/elif blocks.
# Because str is a sequence type, the simplest way to distinguish it from from any other immutable sequence is to do an explicit isinstance(x, str) check.
On the other side, it's usually OK to perform an isinstance check against an ABC if you must enforce an API contract. Outside of frameworks, duck typing is often simpler and more flexible than type checks.
5. Subclassing an ABC
Example 11-8. frenchdeck2.py FrenchDeck2, a subclass of collections.MutableSequence
import collections Card = collections.namedtuple("Card", ["rank", "suit"]) class FrenchDeck2(collections.MutableSequence): ranks = [str(n) for n in range(2, 11)] + list("JQKA") suits = "spades diamonds clubs hearts".split() def __init__(self): self._cards = [Card(rank, suit) for suit in self.suits for rank in self.ranks] def __len__(self): return len(self._cards) def __getitem__(self, position): return self._cards[position] def __setitem__(self, key, value): # __setitem__ is all we need to enable shuffling... self._cards[key] = value def __delitem__(self, key): del self._cards[key] def insert(self, position, value): self._cards.insert(position, value) """ Subclassing MutableSequence forces us to implement __delitem__, an abstract method of that ABC. We are also required to implement insert, the third abstract method of MutableSequence. """ """ Python does not check for the implementation of the abstract methods at import time (when this example module is loaded and compiled), but only at runtime when we try to instantiate FrenchDeck2. Then, if we fail to implement any abstract method, we get a TypeError exception with a message such as "Can't instantiate abstract class FrenchDeck2 with abstract methods __delitem__, insert". That's why we must implement __delitem__ and insert, even if our FrenchDeck2 examples do not need those behaviors: the MutableSequence ABC demands them. """ """ From Sequence, FrenchDeck2 inherits the following ready-to-use concrete methods: __contains__, __iter__ , __reversed__, index, and count. From MutableSequence, it gets append, reverse, extend, pop, remove, and __iadd__ . As the coder of a concrete subclass, you may be able to override methods inherited from ABCs with more efficient implementations. For example, __contains__ works by doing a full scan of the sequence, but if your concrete sequence keeps its items sorted, you can write a faster __contains__ that does a binary search using bisect function. """
6. ABCs in the Standard Library
Since Python 2.6, ABCs are available in the standard library. Most are defined in the collections.abc module, but there are others. You can find ABCs in the numbers and io packages.
Iterable, Container, and Sized
Every collection should either inherit from these ABCs or at least implement compatible protocols. Iterable support iteration with __iter__, Container supports the in operator with __contains__ , and Sized supports len() with __len__ .
Sequence, Mapping, and Set
These are the main immutable collection types, and each has a mutable subclass.
MappingView
In Python 3, the objects returned from the mapping methods .items(), .keys(), and .values() inherit ItemView, KeysView, ValuesView, respectively. The first two also inherit the rich interface of Set.
Callable and Hashable
These ABCs are not so closely related to collections, but collections.abc was the first package to define ABCs in the standard library, and these two were deemed important enough to be included. I've never seen subclass of either Callable or Hashable. Their main use is to support the isinstance built-in as a safe way of determining whether an object is callable or hashable. # For callable detection, there is the callable() built-in function -- but there is no equivalent hashable() function, so isinstance(my_obj, Hashable) is the preferred way to test for a hashable object.
Iterator
Iterator subclass Iterable
After the collections.abc package, the most useful package of ABCs in the standard library is numbers.
7. The Numbers Tower of ABCs
The numbers package defines the so-called "numberic tower", where Number is the top-most superclass, Complex is its immediate subclass, and so on, down to Integral:
- Number
- Complex
- Real
- Rational
- Integral
So if you need to check for an integer, use isinstance(x, numbers.Integral) to accept int, bool (which subclasses int), or other integer types that may be provided by external libraries that register their types with the numbers ABCs.
If, on the other hand, a value can be a floating-point type, you write isinstance(x, numbers.Real), and your code will happily take bool, int, float, fractions.Fraction, or any other non-complex numerical type provided by an external library, such as NumPy, which is suitably registered.
PS: decimal.Decimal is not registered as a virtual subclass of numbers.Real . The reason is that, if you need the precision of Decimal in you program, then you want to be protected from accidental mixing of decimals with other less precise numeric types, particularly floats.
8. Defining and Using an ABC
The Tombola ABC has four methods. The two abstract methods are:
- .load(...) : put items into the container.
- .pick() : remove one item at random from the container, returning it.
The concrete methods are:
- .loaded() : return True if there is at least one item in the container.
- .inspect() : return a sorted tuple built from the items currently in the container, without changing its contents (its internal ordering is not preserved)
Example 11-9. tombola.py: Tombola is an ABC with two abstract methods and two concrete methods.
import abc class Tombola(abc.ABC): # To define an ABC, subclass abc.ABC @abc.abstractmethod def load(self, iterable): # An abstract method is marked with the @abstractmethod decorator, and often its body is empty except for a docstring. """Add items from an iterable""" @abc.abstractmethod def pick(self): # The docstring instructs implementers to raise LookupError if there are no items to pick. """Remove item at random, returning it This method should raise 'LookupError' when the instance is empty. """ def loaded(self): # An ABC may include concrete methods. """Return 'True' if there's at least 1 item, 'False' otherwise""" # Concrete methods in an ABC must reply on the interface defined by the ABC (i.e., other concrete or abstract methods or propterties of the ABC). return bool(self.inspect()) def inspect(self): """Return a sorted tuple with the items currently inside""" items = [] while True: # We can't know how concrete subclass will store the items, but we can build the inspect result by emptying the Tombola with successive calls to .pick() ... try: items.append(self.pick()) except LookupError: break self.load(items) # ... then use .load() to put everything back. return tuple(sorted(items)) """ An abstract method can actually have an implementation. Even if it does, subclass will still be forced to override it, but they will be able to invoke the abstract method with super(), adding functionality to it instead of implementing from scratch. """ # It's OK to provide concrete methods in ABCs, as long as they only depend on other methods in the interface. # The .load() method in this example may not be as silly, but it's expensive: it calls .inspect() to build the sorted tuple just to apply bool() on it.
9. Subclassing the Tombola ABC
Example 11-12. bingo.py: BingoCage is a concrete subclass of Tombola
import random from ex11_9_tombola import Tombola class BingoCage(Tombola): def __init__(self, items): self._randomizer = random.SystemRandom() # random.SystemRandom implements the random API on top of the os.urandom(...) function, which provides random bytes "suitable for cryptographic use". self._items = [] self.load(items) # Delegating initial loading to the .load(...) method. def load(self, items): self._items.extend(items) self._randomizer.shuffle(self._items) # Instead of the plain random.shuffle() function, we use the .shuffle() method of our SytemRandom instance. def pick(self): try: return self._items.pop() except IndexError: raise LookupError("pick from empty BingoCage") def __call__(self): self.pick() # The expensive .loaded() and the silly .inspect() methods could be overridden with much faster one-liners. The pointer here is: we can be lazy and just inherit the suboptimal concrete methods from an ABC.
Example 11-13. lotto.py: LotteryBlower is a concrete subclass that overrides the inspect and loaded methods from Tombola.
import random from ex11_9_tombola import Tombola class LotteryBlower(Tombola): def __init__(self, iterable): self._balls = list(iterable) # The initializer accepts any iterable: the argument is used to build a list. def load(self, iterable): self._balls.extend(iterable) def pick(self): try: position = random.randrange(len(self._balls)) except ValueError: raise LookupError('pick from empty BingoCage') return self._balls.pop(position) def loaded(self): return bool(self._balls) # Override .loaded() to avoid calling .inspect(). We can make it faster by working with self._balls directly -- no need to build a whole sorted tuple. def inspect(self): return tuple(sorted(self._balls)) """ This example illustrates an idiom worth mentioning: in __init__ , self._balls stores list(iterable) and not just a reference (i.e., we did not merely assign iterable to self._balls). This makes our LotteryBlower flexible because the iterable argument may be any iterable type. And we make sure to store its items in a list so we can pop items. And even if we always get lists as the iterable argument, list(iterable) produces a copy of the argument, which is a good practice considering we will be removing items from it and the client may not be expecting the list of items she provided to be changed. """
10. A Virtual Subclass of Tombola
We now come to the crucial dynamic feature of goose typing: declaring virtual subclasses with the register method.
An essential characteristic of goose typing is the ability to register a class as a virtual subclass of an ABC, even if it does not inherit from it. This is done by calling a register method on the ABC. The registered class then becomes a virtual subclass of the ABC, and will be recognized as such by functions like issubclass and isinstance, but it will not inherit any methods or attributes from the ABC.
# Virtual subclasses do not inherit from their registered ABCs, and are not checked for conformance to the ABC interface at any time, not even when they are instantiated. It's up to the subclass to actually implement all the methods needed to avoid runtime errors.
The register method is usually invoked as a plain function, but it can also be used as a decorator.
Example 11-14. tombolist.py: class TomboList is a virtual subclass of Tombola
from random import randrange from ex11_9_tombola import Tombola @Tombola.register # TomboList is registered as a virtual subclass of Tombola. class TomboList(list): # TomboList extends list. def pick(self): if self: # TomboList inherits __bool__ from list, and that returns True if the list is not empty. position = randrange(len(self)) return self.pop(position) # Our pick calls self.pop, inherited from list, passing a random item index. else: raise LookupError('pop from empty TomboList') load = list.extend # TomboList.load is the same as list.extend def loaded(self): return bool(self) # loaded delegates to bool. # the bool built-in function doesn't need __bool__ to work because it cal also use __len__ . def inspect(self): return tuple(sorted(self)) # Tombola.register(TomboList) # If you are using Python 3.3 or earlier, you can't use .register as a class decorator. You must use standard call syntax. # 测试如下： """ >>> from ex11_9_tombola import Tombola >>> from ex11_14_tombolist import TomboList >>> >>> issubclass(TomboList, Tombola) True >>> t = TomboList(range(100)) >>> isinstance(t, Tombola) True """ # __mro__ method basically lists the class and its superclass in the order Python uses to search for methods. If you inspect the __mro__ of TomboList, you'll see that it lists only the "real" superclasses: """ >>> TomboList.__mro__ (<class 'ex11_14_tombolist.TomboList'>, <class 'list'>, <class 'object'>) """ # Tombola is not in TomboList.__mro__ , so TomboList does not inherit any methods from Tombola.
11. How the Tombola Subclass Were Tested
Two class attributes that allow introspection of a class hierarchy:
__subclasses__()
Method that returns a list of the immediate subclass of the class(the direct descendants that are alive in memory). The list does not include virtual subclasses.
_abc_registry
Data attribute -- available only in ABCs -- that is bound to a WeakSet with weak reference to registered virtual subclasses of the abstract class.
12. Usage of register in Practice
Even if register can now be used as a decorator, it's nore widely deployed as a function to register classes defined elsewhere. For example, in the source code for the collections.abc module, the built-in types tuple, str, range, and memoryview are registered as virtual subclasses of Sequence like this:
# Sequence.register(tuple) # Sequence.register(str) # Sequence.register(range) # Sequence.register(memoryview)
Several other built-in types are registered to ABCs in _collections_abc.py. Those registrations happen only when that module is imported.
13. Geese Can Behave as Ducks
A class can be recognized as a virtual subclass of an ABC even without registration.
>>> class Struggle: ... def __len__(self): return 23 ... >>> from collections import abc >>> isinstance(Struggle(), abc.Sized) True >>> issubclass(Struggle, abc.Sized) True # Class Struggle is considered a subclass of abc.Sized by the issubclass function(and, consequently, by isinstance as well) because abc.Sized implements a special class method named __subclasshook__ .
Example 11-17. Sized definition from the source code of Lib/_collections_abc.py (Python 3.6)
class Sized(metaclass=ABCMeta): __slots__ = () @abstractmethod def __len__(self): return 0 @classmethod def __subclasshook__(cls, C): if cls is Sized: return _check_methods(C, "__len__") return NotImplemented def _check_methods(C, *methods): mro = C.__mro__ for method in methods: for B in mro: if method in B.__dict__: # 1. If there is an attribute named __len__ in the __dict__ of any class listed in C.__mro__ (i.e., C and its superclasses) ... if B.__dict__[method] is None: return NotImplemented break else: return NotImplemented # 3. Otherwise return NotImplemented to let the subclass check proceed. return True # 2. ... return True, signaling that C is a virtual subclass of Sized.
14. Is Python Weakly Typed ?
Strong versus Weak Typing
If the language rarely performs implicit conversion of types, it's considered strongly typed; if it often does it, it's weakly typed. Java, C++, and Python are strongly typed. PHP, JavaScript, and Perl are weakly typed.
Static Versus Dynamic Typing
If type-checking is performed at compile time, the language is statically typed; if it happens at runtime, it's dynamically typed. Static typing requires type declarations(some modern languages use type inference to avoid some of that). Fortran and Lisp are the two oldest programming languages still alive and they use, respectively, static and dynamic typing.
Python uses dynamic and strong typing.