annotate piecrust/chefutil.py @ 411:e7b865f8f335

bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.
author Ludovic Chabant <ludovic@chabant.com>
date Fri, 12 Jun 2015 17:09:19 -0700
parents a561fbad0b7f
children 0e9a94b7fdfa
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import time
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 272
diff changeset
2 import logging
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 272
diff changeset
3 import contextlib
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 from colorama import Fore
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 272
diff changeset
7 @contextlib.contextmanager
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 272
diff changeset
8 def format_timed_scope(logger, message, *, level=logging.INFO, colored=True):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 272
diff changeset
9 start_time = time.perf_counter()
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 272
diff changeset
10 yield
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 272
diff changeset
11 logger.log(level, format_timed(start_time, message, colored=colored))
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 272
diff changeset
12
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 272
diff changeset
13
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 99
diff changeset
14 def format_timed(start_time, message, indent_level=0, colored=True):
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 272
diff changeset
15 end_time = time.perf_counter()
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 99
diff changeset
16 indent = indent_level * ' '
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 time_str = '%8.1f ms' % ((end_time - start_time) * 1000.0)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 if colored:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19 return '[%s%s%s] %s' % (Fore.GREEN, time_str, Fore.RESET, message)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 99
diff changeset
20 return '%s[%s] %s' % (indent, time_str, message)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
22
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
23 def log_friendly_exception(logger, ex):
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
24 indent = ''
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
25 while ex:
272
a561fbad0b7f logging: If an error doesn't have a message, print its type.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
26 ex_msg = str(ex)
a561fbad0b7f logging: If an error doesn't have a message, print its type.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
27 if not ex_msg:
a561fbad0b7f logging: If an error doesn't have a message, print its type.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
28 ex_msg = '%s exception was thrown' % type(ex).__name__
a561fbad0b7f logging: If an error doesn't have a message, print its type.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
29 logger.error('%s%s' % (indent, ex_msg))
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
30 indent += ' '
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
31 ex = ex.__cause__
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
32
99
8703be118430 Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
33
8703be118430 Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
34 def print_help_item(s, title, description, margin=4, align=25):
8703be118430 Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
35 s.write(margin * ' ')
8703be118430 Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
36 s.write(title)
8703be118430 Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
37 spacer = (align - margin - len(title) - 1)
8703be118430 Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
38 if spacer <= 0:
8703be118430 Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
39 s.write("\n")
8703be118430 Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
40 s.write(' ' * align)
8703be118430 Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
41 else:
8703be118430 Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
42 s.write(' ' * spacer)
8703be118430 Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
43 s.write(description)
8703be118430 Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
44 s.write("\n")
8703be118430 Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
45