annotate piecrust/sources/prose.py @ 411:e7b865f8f335

bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.
author Ludovic Chabant <ludovic@chabant.com>
date Fri, 12 Jun 2015 17:09:19 -0700
parents 65db6df28120
children 4850f8c21b6e
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
157
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
369
4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 242
diff changeset
3 import copy
157
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import logging
369
4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 242
diff changeset
5 from piecrust.sources.base import MODE_CREATING, MODE_PARSING
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents: 186
diff changeset
6 from piecrust.sources.default import DefaultPageSource
157
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 logger = logging.getLogger(__name__)
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents: 186
diff changeset
12 class ProseSource(DefaultPageSource):
157
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13 SOURCE_NAME = 'prose'
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 def __init__(self, app, name, config):
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16 super(ProseSource, self).__init__(app, name, config)
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 self.config_recipe = config.get('config', {})
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
186
e61fbae61402 sources: Pass any current mode to `_populateMetadata` when finding pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 170
diff changeset
19 def _populateMetadata(self, rel_path, metadata, mode=None):
e61fbae61402 sources: Pass any current mode to `_populateMetadata` when finding pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 170
diff changeset
20 metadata['config'] = self._makeConfig(rel_path, mode)
157
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21
186
e61fbae61402 sources: Pass any current mode to `_populateMetadata` when finding pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 170
diff changeset
22 def _makeConfig(self, rel_path, mode):
369
4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 242
diff changeset
23 c = copy.deepcopy(self.config_recipe)
186
e61fbae61402 sources: Pass any current mode to `_populateMetadata` when finding pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 170
diff changeset
24 if c.get('title') == '%first_line%' and mode != MODE_CREATING:
170
c3831a762bc2 sources: Make the `SimplePageSource` more extensible, fix bugs in `prose` source.
Ludovic Chabant <ludovic@chabant.com>
parents: 157
diff changeset
25 path = os.path.join(self.fs_endpoint_path, rel_path)
369
4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 242
diff changeset
26 try:
4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 242
diff changeset
27 c['title'] = get_first_line(path)
394
65db6df28120 tests: Also mock `open` in Jinja to be able to use templates in bake tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 369
diff changeset
28 except IOError:
369
4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 242
diff changeset
29 if mode == MODE_PARSING:
4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 242
diff changeset
30 raise
157
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 return c
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 def get_first_line(path):
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35 with open(path, 'r') as f:
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 while True:
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 l = f.readline()
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 if not l:
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 break
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 l = l.strip()
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 if not l:
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 continue
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 return l
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 return None
55910ab4bfea First draft of the `prose` page source.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45