Mercurial > piecrust2
annotate piecrust/commands/base.py @ 411:e7b865f8f335
bake: Enable multiprocess baking.
Baking is now done by running a worker per CPU, and sending jobs to them.
This changes several things across the codebase:
* Ability to not cache things related to pages other than the 'main' page
(i.e. the page at the bottom of the execution stack).
* Decouple the baking process from the bake records, so only the main process
keeps track (and modifies) the bake record.
* Remove the need for 'batch page getters' and loading a page directly from
the page factories.
There are various smaller changes too included here, including support for
scope performance timers that are saved with the bake record and can be
printed out to the console. Yes I got carried away.
For testing, the in-memory 'mock' file-system doesn't work anymore, since
we're spawning processes, so this is replaced by a 'tmpfs' file-system which
is saved in temporary files on disk and deleted after tests have run.
author | Ludovic Chabant <ludovic@chabant.com> |
---|---|
date | Fri, 12 Jun 2015 17:09:19 -0700 |
parents | c2ca72fb7f0b |
children | 456db44dcc53 |
rev | line source |
---|---|
0 | 1 import logging |
57
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
2 import argparse |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
3 import functools |
1
aaa8fb7c8918
Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
4 from piecrust.pathutil import SiteNotFoundError |
0 | 5 |
6 | |
7 logger = logging.getLogger(__name__) | |
8 | |
9 | |
10 class CommandContext(object): | |
57
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
11 def __init__(self, app, parser, args): |
1
aaa8fb7c8918
Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
12 self.app = app |
57
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
13 self.parser = parser |
0 | 14 self.args = args |
15 | |
16 | |
17 class ChefCommand(object): | |
18 def __init__(self): | |
19 self.name = '__unknown__' | |
20 self.description = '__unknown__' | |
21 self.requires_website = True | |
371
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
103
diff
changeset
|
22 self.cache_name = 'default' |
0 | 23 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
24 def setupParser(self, parser, app): |
0 | 25 raise NotImplementedError() |
26 | |
27 def run(self, ctx): | |
57
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
28 raise NotImplementedError("Command '%s' doesn't implement the `run` " |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
29 "method." % type(self)) |
0 | 30 |
57
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
31 def checkedRun(self, ctx): |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
32 if ctx.app.root_dir is None and self.requires_website: |
1
aaa8fb7c8918
Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
33 raise SiteNotFoundError() |
57
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
34 return self.run(ctx) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
35 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
36 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
37 class ExtendableChefCommand(ChefCommand): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
38 def __init__(self): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
39 super(ExtendableChefCommand, self).__init__() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
40 self._extensions = None |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
41 |
99
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
42 def getExtensions(self, app): |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
43 self._loadExtensions(app) |
99
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
44 return self._extensions |
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
45 |
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
46 def setupExtensionParsers(self, subparsers, app): |
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
47 for e in self.getExtensions(app): |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
48 p = subparsers.add_parser(e.name, help=e.description) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
49 e.setupParser(p, app) |
99
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
50 p.set_defaults(sub_func=e.checkedRun) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
51 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
52 def _loadExtensions(self, app): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
53 if self._extensions is not None: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
54 return |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
55 self._extensions = [] |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
56 for e in app.plugin_loader.getCommandExtensions(): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
57 if e.command_name == self.name and e.supports(app): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
58 self._extensions.append(e) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
59 |
0 | 60 |
99
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
61 class ChefCommandExtension(object): |
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
62 command_name = '__unknown__' |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
63 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
64 def supports(self, app): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
65 return True |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
66 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
67 |
99
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
68 class HelpCommand(ExtendableChefCommand): |
57
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
69 def __init__(self): |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
70 super(HelpCommand, self).__init__() |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
71 self.name = 'help' |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
72 self.description = "Prints help about PieCrust's chef." |
103
028df35a690e
Fix using `chef` outside of a website.
Ludovic Chabant <ludovic@chabant.com>
parents:
99
diff
changeset
|
73 self.requires_website = False |
99
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
74 self._topic_providers = [] |
57
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
75 |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
76 @property |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
77 def has_topics(self): |
99
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
78 return len(self._topic_providers) > 0 |
57
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
79 |
99
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
80 def getTopics(self): |
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
81 return [(n, d) for (n, d, e) in self._topic_providers] |
57
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
82 |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
83 def setupParser(self, parser, app): |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
84 parser.add_argument('topic', nargs='?', |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
85 help="The command name or topic on which to get help.") |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
86 |
99
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
87 extensions = self.getExtensions(app) |
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
88 for ext in extensions: |
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
89 for name, desc in ext.getHelpTopics(): |
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
90 self._topic_providers.append((name, desc, ext)) |
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
91 |
57
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
92 def run(self, ctx): |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
93 topic = ctx.args.topic |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
94 |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
95 if topic is None: |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
96 ctx.parser.print_help() |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
97 return 0 |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
98 |
99
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
99 for name, desc, ext in self._topic_providers: |
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
100 if name == topic: |
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
101 print(ext.getHelpTopic(topic, ctx.app)) |
8703be118430
Changes to `help` command and extendable commands:
Ludovic Chabant <ludovic@chabant.com>
parents:
57
diff
changeset
|
102 return 0 |
57
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
103 |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
104 for c in ctx.app.plugin_loader.getCommands(): |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
105 if c.name == topic: |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
106 fake = argparse.ArgumentParser( |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
107 prog='%s %s' % (ctx.parser.prog, c.name), |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
108 description=c.description) |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
109 c.setupParser(fake, ctx.app) |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
110 fake.print_help() |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
111 return 0 |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
112 |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
113 raise Exception("No such command or topic: %s" % topic) |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
114 |
c8c522dacfea
Add `help` function, cleanup argument handling.
Ludovic Chabant <ludovic@chabant.com>
parents:
3
diff
changeset
|
115 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
116 class _WrappedCommand(ChefCommand): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
117 def __init__(self, func, name, description): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
118 super(_WrappedCommand, self).__init__() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
119 self.func = func |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
120 self.name = name |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
121 self.description = description |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
122 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
123 def run(self, ctx): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
124 self.func(ctx) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
125 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
126 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
127 def simple_command(f, name, description=None): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
128 @functools.wraps(f) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
129 def wrapper(*args, **kwargs): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
130 return f(*args, **kwargs) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
131 cmd = _WrappedCommand(f, name, description) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
132 f.__command_class__ = cmd |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
133 return wrapper |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
134 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
135 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
136 def get_func_command(f): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
137 return getattr(f, '__command_class__') |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
1
diff
changeset
|
138 |