Function decorators let us "mark" functions in the source code to enhance their behavior in some way. This is powful stuff, but mastering it requires understanding closures.
Aside from their appliation in decorators, closures are also essential for effective asynchronous programming with callbacks, and for coding in a functional style whenever it makes sense.
1. Decorators 101
Example 7-1. A decorator usually replaces a function with a different one
""" A decorator is a callable that takes another function as argument(the decorated function). The decorator may perform some processing with the decorated function, and returns it or replaces it with another function or callable object. """ """ Assuming an existing decorator named decorate, this code: @decorate def target(): print('running target()') Has the same effect as writing this: def target(): print('running target()') target = decorate(target) The end result is the same: at the end of either of these snippets, the target name does not necessarily refer to the original target function, but to whatever function is returned by decorate(target). """ # This example show that the decorated function is replaced. >>> def deco(func): ... def inner(): ... print("running inner()") ... return inner ... >>> @deco ... def target(): ... print('running target()') ... >>> target() # Invoking the decorated target actually runs inner. running inner() >>> target # target is a new reference to inner. <function deco.<locals>.inner at 0x1036259d8> """ Strictly speaking, decorators are just syntactic sugar. As we just saw, you can always simply call a decorator like any regular callable, passing another functions. Sometimes that is actually convenient, especially when doing metaprogramming --- changing program behavior at runtime. """
To Summarize:
# the first crucial fact about decorators is that they have the power to replace the decorated function with a different one. # the second crucial fact is that they are executed immediately when a module is loaded.
2. When Python Execute Decorators
A key feature of decorators is that they run right after the decorated function is defined. This is usually at import time. (i.e., when a module is loaded by Python).
Example 7-2. The ex7_2_registration.py module
registry = [] def register(func): print('running register(%s)' % func) registry.append(func) return func # Return func: we must return a function; here we return the same received as argument. @register def f1(): print('running f1()') @register def f2(): print('running f2()') def f3(): print('running f3()') def main(): print('running main()') print('registry-->', registry) f1() f2() f3() if __name__ == '__main__': main() # main() is only invoked if ex7_2_registration.py runs as a script. # The output of running ex7_2_registration.py as a script looks like this: """ NEO:ch7-function-decorators-closures zhenxink$ python3 ex7_2_registration.py running register(<function f1 at 0x103e61488>) running register(<function f2 at 0x103e61598>) running main() registry--> [<function f1 at 0x103e61488>, <function f2 at 0x103e61598>] running f1() running f2() running f3() """ """ register runs (twice) before any other function in the module. When register is called, it receives as an argument the function object being decorated -- for example, <function f1 at 0x103e61488> . After the module is loaded, the registry holds references to the two decorated functions: f1 and f2. These functions, as well as f3, are only executed when explicitly called by main. """ # If ex7_2_registration.py is imported(and not run as a script), the output is this: """ >>> import ex7_2_registration running register(<function f1 at 0x1035619d8>) running register(<function f2 at 0x103561a60>) >>> ex7_2_registration.registry [<function f1 at 0x1035619d8>, <function f2 at 0x103561a60>] """ """ The main point of Example 7-2 is to emphasize that function decorators are executed as soon as the module is imported, but the decorated functions only run when they are explicitly invoked. This highlights the difference between import time and runtime. (导入时 和 运行时) """ # Considering how decorators are commonly employed in real code, Example 7-2 is unusual in two ways: """ 1. The decorator function is defined in the same module as the decorated functions. A real decorator is usually defined in one module and applied to functions in other modules. 2. The register decorator returns the same function passed as argument. In practice, most decorators define as inner function and return it. """ """ Even though the register decorator in Example 7-2 returns the decorated function unchanged, that technique is not useless. Similar decorators are used in many Python web frameworks to add functions to some central registry -- for example, a registry mapping URL patterns to functions that generate HTTP response. Such registration decorators may or may not change the decorated function. """
3. Decorator-Enhanced Strategy Pattern
Example 7-3. The promos list is filled by the promotion decorator
""" Example 6-6 is the repetition of the function names in their definitions and then in the promos list used by the best_promo function to determine the highest discount applicable. The repetition is problematic because someone may add a new promotional strategy function and forget to manually add it to the promos list -- in which case, best_promo will silently ignore the new strategy, introducing a subtle bug in the system. Example 7-3 solves this problem with a registration decorator. """ promos = [] def promotion(promo_func): # promotion decorator returns promo_func unchanged, after adding it to the promos list. promos.append(promo_func) return promo_func @promotion # Any function decorated by @promotion will be added to promos. def fidelity(order): """5% discount for customers with 1000 or more fidelity points""" return order.total() * .5 if order.customer.fidelity >= 1000 else 0 @promotion def bulk_item(order): """10% discount for each LineItem with 20 or more units""" discount = 0 for item in order.cart: if item.quantity >= 20: discount += item.total() * .1 return discount @promotion def large_promo(order): """7% discount for orders with 10 or more distinct items""" distinct_items = {item.product for item in order.cart} if len(distinct_items) >= 10: return order.total() * .07 return 0 def best_promo(order): """Select best discount available """ return max(promo(order) for promo in promos) """ This solution in this example has several advantages: 1. The promotion strategy functions don't have to use special names.(i.e., they don't need to use the _promo suffix). 2. The @promotion decorator highlights the purpose of the decorated function, and also makes it easy to temporarily disable a promotion: just comment out the decorator. 3. Promotional discount strategies may be defined in other modules, anywhere in the system, as long as the @promotion decorator is applied to them. """ """ Most decorators do change the decorated function. They usually do it by defining an inner function and returning it to replace the decorated function. Code that uses inner functions almost always depends on closures to operate correctly. """
4. Variable Scope Rules
Example 7-5. Variable b is local, because it is assigned a value in the body of the function.
>>> b = 6 >>> def f2(a): ... print(a) ... print(b) ... b = 9 ... >>> f2(3) 3 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in f2 UnboundLocalError: local variable 'b' referenced before assignment """ When Python compiles the body of the function, it decides that b is a local variable because it is assigned within the function. The generated bytecode reflects this decision and will try to fetch b from the local environment. When f2(3) is called and tries to fetch the value of local variable b it discovers that b is unbound. This is not bug, but a design choice: Python does not require you to declare variables, but assumes that a variable assigned in the body of a function is local. """
5. Closures
Example 7-8. average_oo.py: A class to calculate a running average.
""" A closure is a function with an extended scope that encompass nonglobal variables referenced in the body of function but not defined there. And the function can access nonglobal variables taht are defined outside of its body. """ # an avg function to compute the mean of an ever-increasing series of values. class Averager(object): def __init__(self): self.series = [] def __call__(self, new_value): # The Averager class will create instances that are callable. self.series.append(new_value) total = sum(self.series) return total / len(self.series) # 运行结果： """ >>> avg = Averager() >>> avg(10) 10.0 >>> avg(11) 10.5 >>> avg(12) 11.0 >>> """
Example 7-9. average.py: A high-order function to calculate a running average.
"""This example is a functional implementation, using the high-order function make_averager""" def make_averager(): series = [] def averager(new_value): series.append(new_value) total = sum(series) return total / len(series) return averager """ When invoked, make_averager returns an averager function object. Each time an averager is called, it appends the passed argument to the series, and compute the current average. """ # 运行结果： """ >>> avg = make_averager() >>> avg(10) 10.0 >>> avg(11) 10.5 >>> avg(12) 11.0 >>> """ """ Note the similarities of these two examples: we call Averager() or make_averager() to get a callable object avg that will update the historical series and calculate the current mean. In Example 7-8, avg is an instance of Averager, and in Example 7-9 it is the inner function, averager. Either way, we just call avg(n) to include n in the series and get the updated mean. """ # Within averager, series is a free variable （自由变量）. This is a technical term meaning a variable that is not bound # in the local scope. # The closure for averager extends the scope of that function to include the binding for the free variable -- series. # Inspecting the returned averager object below show how Python keeps the names of local and free variables in the # __code__ attribute that represents the compiled body of the function. # Example 7-11: Inspecting the function created by make averager in Example 7-9. """ >>> avg.__code__.co_varnames ('new_value', 'total') >>> avg.__code__.co_freevars ('series',) >>> """ """ The binding for series is kept in the __closure__ attribute of the returned function avg. Each item in avg.__closure__ corresponds to a name in avg.__code__.co_freevars. These items are cells, and they have an attribute called cell_contents where the actual value can be found. """ # Example 7-12. """ >>> avg.__code__.co_freevars ('series',) >>> avg.__closure__ (<cell at 0x1032d0fa8: list object at 0x10332f608>,) >>> avg.__closure__[0].cell_contents [10, 11, 12] """
To summarize:
# A closure is a function that retains the bindings of the free variables that exist when the function is defined, so that they can be used later when the function is invoked and the defining scope is no longer available.
6. The nonlocal Declaration
Our previous impementation of make_averager was not efficient. In Example 7-9, we stored all the values in the historical series and computed their sum every time averager was called. A better implementation would just store the total and the numbers of items so far, and compute the mean of these two numbers.
Example 7-13. A broken high-order function to calculate a running average without keeping all history.
"""This example is a broken implementation""" def make_averager(): count = 0 total = 0 def averager(new_value): count += 1 total += new_value return total / count return averager # If you try Example, here is what you get: """ >>> avg = make_averager() >>> avg(10) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "......../ex7_13_broken_high_order_func.py", line 9, in averager count += 1 UnboundLocalError: local variable 'count' referenced before assignment """ """ The problem is that the statement count += 1 actually means the same as count = count + 1 , when count is a number or any immutable type. So we are actually assigning to count in the body of averager, and that makes it a local variable. The same problem affects the total variable. We did not have this problem in Example 7-9 because we never assigned to the series name; we only called series.append and invoke sum and len on it. So we took advantage of the fact that lists are mutable. But with immutable types like numbers, strings, tuples, etc., all you can do is read, but never update. If you try to rebind them, as in count = count + 1, then you are implicitly creating a local variable count. It is no longer a free variable, and therefore it is not saved in the closure. To work around this, the nonlocal declaration was introduced in Python 3. It lets you flag a variable as a free variable even when it is assigned a new value within the function. If a new value is assigned to a nonlocal variable, the binding stored in the closure is changed. A correct implementation of our newest make_averager looks like Example 7-14 """
Example 7-14. Calculate a running average without keeping all history (fixed with the use of nonlocal)
def make_averager(): count = 0 total = 0 def averager(new_value): nonlocal count, total count += 1 total += new_value return total / count return averager """ >>> avg = make_averager() >>> >>> avg(10) 10.0 >>> avg(11) 10.5 >>> avg(12) 11.0 """
7. Implementing a Simple Decorator
Example 7-15. A simple decorator to output the running time of functions.
""" This example is a decorator that clocks every invocation of the decorated function and prints the elapsed time, the arguments passed, and the result of the call. """ import time def clock(func): def clocked(*args): t0 = time.perf_counter() result = func(*args) # the func is a free variable. elapsed = time.perf_counter() - t0 name = func.__name__ arg_str = ",".join(arg for arg in args) print('[%0.8fs] %s(%s) -> %r' % (elapsed, name, arg_str, result)) return result return clocked
Example 7-16. Using the clock decorator
"""This example demonstrates the use of the clock decorator""" import time from ex7_15_clockdeco import clock @clock def snooze(seconds): time.sleep(seconds) @clock def factorial(n): return 1 if n < 2 else n * factorial(n - 1) if __name__ == "__main__": print('*' * 40, 'Calling snooze(.123)') snooze(.123) print('*' * 40, 'Calling factorial(6)') print('6! =', factorial(6)) # 运行结果： """ NEO:ch7-function-decorators-closures zhenxink$ python3 ex7_16_clockdemo_demo.py **************************************** Calling snooze(.123) [0.12794940s] snooze(0.123) -> None **************************************** Calling factorial(6) [0.00000257s] factorial(1) -> 1 [0.00004032s] factorial(2) -> 2 [0.00005923s] factorial(3) -> 6 [0.00007498s] factorial(4) -> 24 [0.00009234s] factorial(5) -> 120 [0.00011269s] factorial(6) -> 720 6! = 720 """ """ How It Work: Remember that this code: @clock def factorial(n): return 1 if n < 2 else n * factorial(n-1) Actually does this: def factorial(n): return 1 if n < 2 else n * factorial(n-1) factorial = clock(factorial) So, in both examples, clock gets the factorial function as its func argument. It then creates and returns the clocked function, which the Python interpreter assign to factorial behind the scenes(Python解释器把 clocked 函数对象赋值给了 factorial函数). In fact, if you import this clockdeco_demo module and check the __name__ of the factorial, this is what you get: >>> import ex7_16_clockdemo_demo >>> ex7_16_clockdemo_demo.factorial.__name__ 'clocked' >>> So factorial now actually holds a reference to the clocked function. From now on, each time factorial(n) is called, clocked(n) gets executed. This is the typical behavior of a decorator: it replaces the decorated function with a new function that accepts the same arguments and (usually) returns whatever the decorated function was supposed to return, while also doing some extra processing. """
Example 7-17. An improved clock decorator
""" The clock decorator implemented in Example 7-15 has a few shortcomings: it does not support keyword arguments, and it mask the __name__ and __doc__ of the decorated function. This example uses the functools.wraps decorator to copy the relevant attributes from func to clocked. """ import time import functools def clock(func): @functools.wraps(func) def clocked(*args, **kwargs): t0 = time.perf_counter() result = func(*args, **kwargs) elapsed = time.perf_counter() - t0 name = func.__name__ arg_lst = [] if args: arg_lst.append(", ".join(repr(arg) for arg in args)) if kwargs: pairs = ["%s=%r" % (k, v) for k, v in sorted(kwargs.items())] arg_lst.append(", ".join(pairs)) arg_str = ", ".join(arg_lst) print('[%0.8fs] %s(%s) -> %r' % (elapsed, name, arg_str, result)) return result return clocked
8. Memoization with functools.lru_cach
functools.lru_cache implements memoization: an optimization technique that works by saving the results fo previous invocations of an expensive function, avoiding repeat computations on previously used arguments. The letters LRU stand for Least Recent Used, meaning that the growth of the cache is limited by discarding the entries that have not been read for a while.
Example 7-18. The very costly recursive way to compute the nth number in the Fibonacci series.
""" The very costly recursive way to compute the nth number in the Fibonacci series. """ from ex7_15_clockdeco import clock @clock def fibonacci(n): if n < 2: return n return fibonacci(n-2) + fibonacci(n-1) if __name__ == "__main__": print(fibonacci(6)) # Here is the result of running this example. Except for the last line, all output is generated by the clock decorator: """ NEO:ch7-function-decorators-closures zhenxink$ python3 ex7_18_recursive_fibo.py [0.00000048s] fibonacci(0) -> 0 [0.00000062s] fibonacci(1) -> 1 [0.00008468s] fibonacci(2) -> 1 [0.00000035s] fibonacci(1) -> 1 [0.00000038s] fibonacci(0) -> 0 [0.00000037s] fibonacci(1) -> 1 [0.00001216s] fibonacci(2) -> 1 [0.00002379s] fibonacci(3) -> 2 [0.00012058s] fibonacci(4) -> 3 [0.00000036s] fibonacci(1) -> 1 [0.00000032s] fibonacci(0) -> 0 [0.00000039s] fibonacci(1) -> 1 [0.00001185s] fibonacci(2) -> 1 [0.00002309s] fibonacci(3) -> 2 [0.00000032s] fibonacci(0) -> 0 [0.00000042s] fibonacci(1) -> 1 [0.00001187s] fibonacci(2) -> 1 [0.00000030s] fibonacci(1) -> 1 [0.00000043s] fibonacci(0) -> 0 [0.00000037s] fibonacci(1) -> 1 [0.00001193s] fibonacci(2) -> 1 [0.00002325s] fibonacci(3) -> 2 [0.00004617s] fibonacci(4) -> 3 [0.00008027s] fibonacci(5) -> 5 [0.00021297s] fibonacci(6) -> 8 8 """ """ The waste is obvious: fibonacci(1) is called eight times, fibonacci(2) five times, etc. But if we just add two lines to use lru_cache, performance is much improved. """
Example 7-19. Faster implementation using caching.
"""Faster implementation using caching""" import functools from ex7_15_clockdeco import clock @functools.lru_cache() # lru_cache must be invoked as a regular function -- note the parentheses in the line: @functools.lru_cache(). The reason is that it accepts configuration parameters. @clock def fibonacci(n): if n < 2: return n return fibonacci(n - 2) + fibonacci(n - 1) # This is an example of stacked decorators: @lru_cache() is applied on the function returned by @clock if __name__ == "__main__": print(fibonacci(6)) # 运行结果： # Execution time is halved, and the function is called only once for each value of n """ NEO:ch7-function-decorators-closures zhenxink$ python3 ex7_19_fibo_cache.py [0.00000043s] fibonacci(0) -> 0 [0.00000060s] fibonacci(1) -> 1 [0.00008756s] fibonacci(2) -> 1 [0.00000089s] fibonacci(3) -> 2 [0.00009955s] fibonacci(4) -> 3 [0.00000060s] fibonacci(5) -> 5 [0.00011164s] fibonacci(6) -> 8 8 """
lru_cache 的用法：
""" lru_cache can be funed by passing two optional arguments. Its full signature is: functools.lru_cache(maxsize=128, typed=False) The maxsize argument determines how many call results are stored. After the cache is full, order results are discarded to make room. For optimal performance, maxsize should be a power of 2. The typed argument, if set to True, stores results of different argument types separately, i.e., distinguishing between float and integer arguments that are normally considered equal, like 1 and 1.0. By the way, because lru_cache uses a dict to store the results, and the keys are made from the positional and keyword arguments used in the calls, all the arguments taken by the decorated function must be hashable. """
9. Generic Functions with Single Dispatch
Example 7-21. singledispatch creates a custom htmlize.register to bundle several functions into a generic function.
""" Because we don't have method or function overloading in Python, we can't create variations of htmlize with different signatures for each data type we want to handle differently. A common solution in Python would be to turn htmlize into a dispatch function, with a chain of if/elif/elif calling specialized functions like htmlize_str, htmlize_int, tec. This is not extensible by users of our module, and is unwieldy: over time, the htmlize dispatcher would become too big, and the coupling between it and the specialized functions would be very tight. The new functools.singledispatch decorator in Python 3.4 allows each module to contribute to the overall solution, and lets you easily provide a specialized function even for classes that you can't edit. If you decorate a plain function with @singledispatch, it becomes a generic function （泛型函数）: a group of functions to perform the same operation in different ways, depending on the type of the first argument. """ from functools import singledispatch from collections import abc import numbers import html @singledispatch # @singledispatch marks the base function that handles the object type. def htmlize(obj): content = html.escape(repr(obj)) return '<pre>{}</pre>'.format(content) @htmlize.register(str) # Each specialized function is decorated with @<<base_function>>.register(<<type>>). def _(text): # The name of the specialized functions is irrelevant; _ is a good choice to make this clear. content = html.escape(text).replace(' ', '<br> ') return '<p>{0}</p>'.format(content) @htmlize.register(numbers.Integral) def _(n): return '<pre>{0} (0x{0:x})</pre>'.format(n) @htmlize.register(tuple) # You can stack several register decorators to support different types with the same function. @htmlize.register(abc.MutableSequence) def _(seq): inner = '</li> <li>'.join(htmlize(item) for item in seq) return '<ul> <li>' + inner + '</li> <ul>' """ When possible, register the specialized functions to handle ABCs(abstract classes) such as numbers.Integral and abc.MutableSequence instead of concrete implementations like int and list. Using ABCs ofr type checking allows your code to support existing or future classes that are either actual or virtual subclasses of those ABCs. A notable quality of the singledispatch mechanism is that you can register specialized functions anywhere in the system, in any module. The advantage of @singledispatch is supporting modular extension: such module can register a specialized function for each type it supports. """
python 重载和泛型函数参考链接：
https://blog.csdn.net/zjbyough/article/details/97152241
10. Stacked Decorators
When two decorators @d1 and @d2 are applied to a function f in that order, the result is the same as f = d1(d2(f))
""" In other words, this: @d1 @d2 def f(): pass Is the same as: def f(): pass f = d1(d2(f)) """
11. Parameterized Decorators
When parsing a decorator in source code, Python takes the decorated function and passes it as the first argument to the decorator function. So how do you make a decorator accept other arguments ? This answer is:
# make a decorator factory that takes those arguments and returns a decorator, which is then applied to the function to be decorated.
11.1 A Parameterized Registration Decorator
Example 7-23. To accept parameters, the new register decorator must be called as a function.
""" Conceptually this register function is not a decorator but a decorator factory. When called, it returns the actual decorator that will be applied to the target function. """ registry = set() def register(active=True): def decorate(func): # The decorate inner function is the actual decorator. print('running register(active=%s) ->decorate(%s)' % (active, func)) if active: registry.add(func) else: registry.discard(func) return func # Because decorate is a decorate, it must return a function （装饰器要返回一个函数） return decorate # register is a decorator factory, so it returns decorate. @register(active=False) # The @register factory must be invoked as a function, with the desired parameters. def f1(): print('running f1()') @register() def f2(): print('running f2()') def f3(): print('running f3()') # The main point is that register() returns decorate, which is then applied to the decorated function. # If we import the module of this example, this is what we get: """ >>> import ex7_23_parametered_regis_deco running register(active=False) ->decorate(<function f1 at 0x102c26a60>) running register(active=True) ->decorate(<function f2 at 0x102c26ae8>) >>> ex7_23_parametered_regis_deco.registry {<function f2 at 0x102c26ae8>} >>> """ # If, instead of using the @ syntax, we used register as a regular function, the syntax needed to decorate a function f # would be register()(f) to add f to the registry or register(active=False)(f) to not add it(or remove it). """ >>> from ex7_23_parametered_regis_deco import * running register(active=False) ->decorate(<function f1 at 0x103d2da60>) running register(active=True) ->decorate(<function f2 at 0x103d2dae8>) >>> registry {<function f2 at 0x103d2dae8>} >>> register()(f3) # The register() expression returns decorate, which is then applied to f3. running register(active=True) ->decorate(<function f3 at 0x103d2d9d8>) <function f3 at 0x103d2d9d8> >>> registry {<function f3 at 0x103d2d9d8>, <function f2 at 0x103d2dae8>} >>> register(active=False)(f2) running register(active=False) ->decorate(<function f2 at 0x103d2dae8>) <function f2 at 0x103d2dae8> >>> registry {<function f3 at 0x103d2d9d8>} >>> """
11.2 The Parameterized Clock Decorator
import time DEFAULT_FMT = '[{elapsed:0.8f}s] {name}({args}) -> {result}' def clock(fmt=DEFAULT_FMT): # clock is the parameterized decorator factory. def decorate(func): # decorate is the actual decorator. def clocked(*_args): t0 = time.perf_counter() _result = func(*_args) # _result is the actual result of the decorated function. elapsed = time.perf_counter() - t0 name = func.__name__ args = ', '.join(repr(arg) for arg in _args) result = repr(_result) # result is the str representation of _result, for display. print(fmt.format(**locals())) # Using **locals() here allows any local variable of clocked to be referenced in the fmt. return _result return clocked return decorate if __name__ == '__main__': @clock() def snooze(seconds): time.sleep(seconds) for i in range(3): snooze(.123) # 运行结果： """ NEO:ch7-function-decorators-closures zhenxink$ python3 ex7_25_param_clock_deco.py [0.12548879s] snooze(0.123) -> None [0.12756666s] snooze(0.123) -> None [0.12711942s] snooze(0.123) -> None """ # Example 7-26. clockdemo_param_demo1.py """ import time from ex7_25_param_clock_deco import clock @clock('{name}: {elapsed}s') def snooze(seconds): time.sleep(seconds) for i in range(3): snooze(.123) """ # 运行结果： """ snooze: 0.12563029094599187s snooze: 0.12794885295443237s snooze: 0.12757847597822547s """ # Decorators are best coded as classed in implementing __call__ , and not as functions like the examples in this chapter.
装饰器类示例：
https://www.cnblogs.com/6min/p/11189610.html
Mastering closures and nonlocal is valuable not only to build decorators, but also to code event-orientied programs for GUIs or asynchronous I/O.