annotate piecrust/sources/array.py @ 411:e7b865f8f335

bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.
author Ludovic Chabant <ludovic@chabant.com>
date Fri, 12 Jun 2015 17:09:19 -0700
parents f130365568ff
children ab5c6a8ae90a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 from piecrust.sources.base import PageSource
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 from piecrust.sources.mixins import SimplePaginationSourceMixin
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 242
diff changeset
3 from piecrust.sources.pageref import PageRef
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 class CachedPageFactory(object):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7 """ A `PageFactory` (in appearance) that already has a page built.
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 """
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 def __init__(self, page):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10 self._page = page
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12 @property
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13 def rel_path(self):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 return self._page.rel_path
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16 @property
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 def metadata(self):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 return self._page.source_metadata
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 @property
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 def ref_spec(self):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22 return self._page.ref_spec
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 @property
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 def path(self):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 return self._page.path
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 def buildPage(self):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29 return self._page
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 class ArraySource(PageSource, SimplePaginationSourceMixin):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 def __init__(self, app, inner_source, name='array', config=None):
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 242
diff changeset
34 super(ArraySource, self).__init__(app, name, config)
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35 self.inner_source = inner_source
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 @property
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 def page_count(self):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 return len(self.inner_source)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 def getPageFactories(self):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 for p in self.inner_source:
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 yield CachedPageFactory(p)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 242
diff changeset
45 def getTaxonomyPageRef(self, tax_name):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 242
diff changeset
46 return None
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 242
diff changeset
47