Mercurial > piecrust2
annotate piecrust/baking/baker.py @ 415:0e9a94b7fdfa
bake: Improve bake record information.
* Store things in the bake record that require less interaction between the
master process and the workers. For instance, don't store the paginator
object in the render pass info -- instead, just store whether pagination
was used, and whether it had more items.
* Simplify information passing between workers and bake passes by saving the
rendering info to the JSON cache. This means the "render first sub" job
doesn't have to return anything except errors now.
* Add more performance counter info.
author | Ludovic Chabant <ludovic@chabant.com> |
---|---|
date | Sat, 20 Jun 2015 19:23:16 -0700 |
parents | e7b865f8f335 |
children | 4a43d7015b75 |
rev | line source |
---|---|
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
1 import copy |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
2 import time |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
3 import os.path |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
4 import queue |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
5 import hashlib |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
6 import logging |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
7 import multiprocessing |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
8 from piecrust.baking.records import ( |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
9 BakeRecordEntry, TransitionalBakeRecord, TaxonomyInfo) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
10 from piecrust.baking.worker import ( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
11 BakeWorkerJob, LoadJobPayload, RenderFirstSubJobPayload, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
12 BakeJobPayload, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
13 JOB_LOAD, JOB_RENDER_FIRST, JOB_BAKE) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
14 from piecrust.chefutil import ( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
15 format_timed_scope, format_timed) |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
16 from piecrust.sources.base import ( |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
17 REALM_NAMES, REALM_USER, REALM_THEME) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
18 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
19 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
20 logger = logging.getLogger(__name__) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
21 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
22 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
23 class Baker(object): |
205
e725af1d48fb
bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents:
199
diff
changeset
|
24 def __init__(self, app, out_dir, force=False): |
127
bc63dc20baa0
Fix how we pass the out directory to the baking modules.
Ludovic Chabant <ludovic@chabant.com>
parents:
120
diff
changeset
|
25 assert app and out_dir |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
26 self.app = app |
127
bc63dc20baa0
Fix how we pass the out directory to the baking modules.
Ludovic Chabant <ludovic@chabant.com>
parents:
120
diff
changeset
|
27 self.out_dir = out_dir |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
28 self.force = force |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
29 self.num_workers = app.config.get('baker/workers', |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
30 multiprocessing.cpu_count()) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
31 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
32 # Remember what taxonomy pages we should skip |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
33 # (we'll bake them repeatedly later with each taxonomy term) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
34 self.taxonomy_pages = [] |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
35 logger.debug("Gathering taxonomy page paths:") |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
36 for tax in self.app.taxonomies: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
37 for src in self.app.sources: |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
38 tax_page_ref = tax.getPageRef(src) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
39 for path in tax_page_ref.possible_paths: |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
40 self.taxonomy_pages.append(path) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
41 logger.debug(" - %s" % path) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
42 |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
43 # Register some timers. |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
44 self.app.env.registerTimer('LoadJob', raise_if_registered=False) |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
45 self.app.env.registerTimer('RenderFirstSubJob', |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
46 raise_if_registered=False) |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
47 self.app.env.registerTimer('BakeJob', raise_if_registered=False) |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
48 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
49 def bake(self): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
50 logger.debug(" Bake Output: %s" % self.out_dir) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
51 logger.debug(" Root URL: %s" % self.app.config.get('site/root')) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
52 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
53 # Get into bake mode. |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
54 start_time = time.perf_counter() |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
55 self.app.config.set('baker/is_baking', True) |
32
43091c9837bf
Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents:
25
diff
changeset
|
56 self.app.env.base_asset_url_format = '%uri%' |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
57 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
58 # Make sure the output directory exists. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
59 if not os.path.isdir(self.out_dir): |
5 | 60 os.makedirs(self.out_dir, 0o755) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
61 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
62 # Load/create the bake record. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
63 record = TransitionalBakeRecord() |
45
efd0d3bacc9e
Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
42
diff
changeset
|
64 record_cache = self.app.cache.getCache('baker') |
333
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
65 record_id = hashlib.md5(self.out_dir.encode('utf8')).hexdigest() |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
66 record_name = record_id + '.record' |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
67 previous_record_path = None |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
68 if not self.force and record_cache.has(record_name): |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
69 with format_timed_scope(logger, "loaded previous bake record", |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
70 level=logging.DEBUG, colored=False): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
71 previous_record_path = record_cache.getCachePath(record_name) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
72 record.loadPrevious(previous_record_path) |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
73 record.current.success = True |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
74 |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
75 # Figure out if we need to clean the cache because important things |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
76 # have changed. |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
77 self._handleCacheValidity(record) |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
78 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
79 # Pre-create all caches. |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
80 for cache_name in ['app', 'baker', 'pages', 'renders']: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
81 self.app.cache.getCache(cache_name) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
82 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
83 # Gather all sources by realm -- we're going to bake each realm |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
84 # separately so we can handle "overriding" (i.e. one realm overrides |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
85 # another realm's pages, like the user realm overriding the theme |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
86 # realm). |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
87 sources_by_realm = {} |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
88 for source in self.app.sources: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
89 srclist = sources_by_realm.setdefault(source.realm, []) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
90 srclist.append(source) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
91 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
92 # Create the worker processes. |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
93 pool = self._createWorkerPool() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
94 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
95 # Bake the realms. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
96 realm_list = [REALM_USER, REALM_THEME] |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
97 for realm in realm_list: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
98 srclist = sources_by_realm.get(realm) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
99 if srclist is not None: |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
100 self._bakeRealm(record, pool, realm, srclist) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
101 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
102 # Bake taxonomies. |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
103 self._bakeTaxonomies(record, pool) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
104 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
105 # All done with the workers. |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
106 self._terminateWorkerPool(pool) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
107 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
108 # Get the timing information from the workers. |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
109 record.current.timers = {} |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
110 for _ in range(len(pool.workers)): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
111 try: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
112 timers = pool.results.get(True, 0.1) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
113 except queue.Empty: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
114 logger.error("Didn't get timing information from all workers.") |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
115 break |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
116 |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
117 for name, val in timers['data'].items(): |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
118 main_val = record.current.timers.setdefault(name, 0) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
119 record.current.timers[name] = main_val + val |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
120 |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
121 # Delete files from the output. |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
122 self._handleDeletetions(record) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
123 |
333
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
124 # Backup previous records. |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
125 for i in range(8, -1, -1): |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
126 suffix = '' if i == 0 else '.%d' % i |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
127 record_path = record_cache.getCachePath( |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
128 '%s%s.record' % (record_id, suffix)) |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
129 if os.path.exists(record_path): |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
130 record_path_next = record_cache.getCachePath( |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
131 '%s.%s.record' % (record_id, i + 1)) |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
132 if os.path.exists(record_path_next): |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
133 os.remove(record_path_next) |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
134 os.rename(record_path, record_path_next) |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
135 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
136 # Save the bake record. |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
137 with format_timed_scope(logger, "saved bake record.", |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
138 level=logging.DEBUG, colored=False): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
139 record.current.bake_time = time.time() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
140 record.current.out_dir = self.out_dir |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
141 record.saveCurrent(record_cache.getCachePath(record_name)) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
142 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
143 # All done. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
144 self.app.config.set('baker/is_baking', False) |
151
fd146f54bdaa
Forgot this wasn't C++.
Ludovic Chabant <ludovic@chabant.com>
parents:
150
diff
changeset
|
145 logger.debug(format_timed(start_time, 'done baking')) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
146 |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
147 return record.detach() |
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
148 |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
149 def _handleCacheValidity(self, record): |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
150 start_time = time.perf_counter() |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
151 |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
152 reason = None |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
153 if self.force: |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
154 reason = "ordered to" |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
155 elif not self.app.config.get('__cache_valid'): |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
156 # The configuration file was changed, or we're running a new |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
157 # version of the app. |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
158 reason = "not valid anymore" |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
159 elif (not record.previous.bake_time or |
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
160 not record.previous.hasLatestVersion()): |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
161 # We have no valid previous bake record. |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
162 reason = "need bake record regeneration" |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
163 else: |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
164 # Check if any template has changed since the last bake. Since |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
165 # there could be some advanced conditional logic going on, we'd |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
166 # better just force a bake from scratch if that's the case. |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
167 max_time = 0 |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
168 for d in self.app.templates_dirs: |
42
9e058e221108
Fix a crash when checking for timestamps on template files.
Ludovic Chabant <ludovic@chabant.com>
parents:
39
diff
changeset
|
169 for dpath, _, filenames in os.walk(d): |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
170 for fn in filenames: |
42
9e058e221108
Fix a crash when checking for timestamps on template files.
Ludovic Chabant <ludovic@chabant.com>
parents:
39
diff
changeset
|
171 full_fn = os.path.join(dpath, fn) |
9e058e221108
Fix a crash when checking for timestamps on template files.
Ludovic Chabant <ludovic@chabant.com>
parents:
39
diff
changeset
|
172 max_time = max(max_time, os.path.getmtime(full_fn)) |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
173 if max_time >= record.previous.bake_time: |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
174 reason = "templates modified" |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
175 |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
176 if reason is not None: |
85
3471ffa059b2
Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
50
diff
changeset
|
177 # We have to bake everything from scratch. |
371
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
369
diff
changeset
|
178 self.app.cache.clearCaches(except_names=['app']) |
45
efd0d3bacc9e
Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
42
diff
changeset
|
179 self.force = True |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
180 record.incremental_count = 0 |
96
0445a2232de7
Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
91
diff
changeset
|
181 record.clearPrevious() |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
182 logger.info(format_timed( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
183 start_time, |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
184 "cleaned cache (reason: %s)" % reason)) |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
185 else: |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
186 record.incremental_count += 1 |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
187 logger.debug(format_timed( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
188 start_time, "cache is assumed valid", |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
189 colored=False)) |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
190 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
191 def _bakeRealm(self, record, pool, realm, srclist): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
192 start_time = time.perf_counter() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
193 try: |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
194 record.current.baked_count[realm] = 0 |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
195 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
196 all_factories = [] |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
197 for source in srclist: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
198 factories = source.getPageFactories() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
199 all_factories += [f for f in factories |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
200 if f.path not in self.taxonomy_pages] |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
201 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
202 self._loadRealmPages(record, pool, all_factories) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
203 self._renderRealmPages(record, pool, all_factories) |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
204 self._bakeRealmPages(record, pool, realm, all_factories) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
205 finally: |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
206 page_count = record.current.baked_count[realm] |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
207 logger.info(format_timed( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
208 start_time, |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
209 "baked %d %s pages." % |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
210 (page_count, REALM_NAMES[realm].lower()))) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
211 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
212 def _loadRealmPages(self, record, pool, factories): |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
213 logger.debug("Loading %d realm pages..." % len(factories)) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
214 with format_timed_scope(logger, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
215 "loaded %d pages" % len(factories), |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
216 level=logging.DEBUG, colored=False, |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
217 timer_env=self.app.env, |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
218 timer_category='LoadJob'): |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
219 for fac in factories: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
220 job = BakeWorkerJob( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
221 JOB_LOAD, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
222 LoadJobPayload(fac)) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
223 pool.queue.put_nowait(job) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
224 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
225 def _handler(res): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
226 # Create the record entry for this page. |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
227 record_entry = BakeRecordEntry(res.source_name, res.path) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
228 record_entry.config = res.config |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
229 if res.errors: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
230 record_entry.errors += res.errors |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
231 record.current.success = False |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
232 self._logErrors(res.path, res.errors) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
233 record.addEntry(record_entry) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
234 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
235 self._waitOnWorkerPool( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
236 pool, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
237 expected_result_count=len(factories), |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
238 result_handler=_handler) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
239 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
240 def _renderRealmPages(self, record, pool, factories): |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
241 logger.debug("Rendering %d realm pages..." % len(factories)) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
242 with format_timed_scope(logger, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
243 "prepared %d pages" % len(factories), |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
244 level=logging.DEBUG, colored=False, |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
245 timer_env=self.app.env, |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
246 timer_category='RenderFirstSubJob'): |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
247 expected_result_count = 0 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
248 for fac in factories: |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
249 record_entry = record.getCurrentEntry(fac.path) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
250 if record_entry.errors: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
251 logger.debug("Ignoring %s because it had previous " |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
252 "errors." % fac.ref_spec) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
253 continue |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
254 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
255 # Make sure the source and the route exist for this page, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
256 # otherwise we add errors to the record entry and we'll skip |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
257 # this page for the rest of the bake. |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
258 source = self.app.getSource(fac.source.name) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
259 if source is None: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
260 record_entry.errors.append( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
261 "Can't get source for page: %s" % fac.ref_spec) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
262 logger.error(record_entry.errors[-1]) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
263 continue |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
264 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
265 route = self.app.getRoute(fac.source.name, fac.metadata, |
363
dd25bd3ce1f9
serve: Refactoring and fixes to be able to serve taxonomy pages.
Ludovic Chabant <ludovic@chabant.com>
parents:
338
diff
changeset
|
266 skip_taxonomies=True) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
267 if route is None: |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
268 record_entry.errors.append( |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
269 "Can't get route for page: %s" % fac.ref_spec) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
270 logger.error(record_entry.errors[-1]) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
271 continue |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
272 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
273 # All good, queue the job. |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
274 job = BakeWorkerJob( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
275 JOB_RENDER_FIRST, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
276 RenderFirstSubJobPayload(fac)) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
277 pool.queue.put_nowait(job) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
278 expected_result_count += 1 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
279 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
280 def _handler(res): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
281 entry = record.getCurrentEntry(res.path) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
282 if res.errors: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
283 entry.errors += res.errors |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
284 record.current.success = False |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
285 self._logErrors(res.path, res.errors) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
286 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
287 self._waitOnWorkerPool( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
288 pool, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
289 expected_result_count=expected_result_count, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
290 result_handler=_handler) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
291 |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
292 def _bakeRealmPages(self, record, pool, realm, factories): |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
293 logger.debug("Baking %d realm pages..." % len(factories)) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
294 with format_timed_scope(logger, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
295 "baked %d pages" % len(factories), |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
296 level=logging.DEBUG, colored=False, |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
297 timer_env=self.app.env, |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
298 timer_category='BakeJob'): |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
299 expected_result_count = 0 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
300 for fac in factories: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
301 if self._queueBakeJob(record, pool, fac): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
302 expected_result_count += 1 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
303 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
304 def _handler(res): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
305 entry = record.getCurrentEntry(res.path, res.taxonomy_info) |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
306 entry.subs = res.sub_entries |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
307 if res.errors: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
308 entry.errors += res.errors |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
309 self._logErrors(res.path, res.errors) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
310 if entry.has_any_error: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
311 record.current.success = False |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
312 if entry.was_any_sub_baked: |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
313 record.current.baked_count[realm] += 1 |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
314 record.dirty_source_names.add(entry.source_name) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
315 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
316 self._waitOnWorkerPool( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
317 pool, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
318 expected_result_count=expected_result_count, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
319 result_handler=_handler) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
320 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
321 def _bakeTaxonomies(self, record, pool): |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
322 logger.debug("Baking taxonomy pages...") |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
323 with format_timed_scope(logger, 'built taxonomy buckets', |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
324 level=logging.DEBUG, colored=False): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
325 buckets = self._buildTaxonomyBuckets(record) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
326 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
327 start_time = time.perf_counter() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
328 page_count = self._bakeTaxonomyBuckets(record, pool, buckets) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
329 logger.info(format_timed(start_time, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
330 "baked %d taxonomy pages." % page_count)) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
331 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
332 def _buildTaxonomyBuckets(self, record): |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
333 # Let's see all the taxonomy terms for which we must bake a |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
334 # listing page... first, pre-populate our big map of used terms. |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
335 # For each source name, we have a list of taxonomies, and for each |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
336 # taxonomies, a list of terms, some being 'dirty', some used last |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
337 # time, etc. |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
338 buckets = {} |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
339 tax_names = [t.name for t in self.app.taxonomies] |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
340 source_names = [s.name for s in self.app.sources] |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
341 for sn in source_names: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
342 source_taxonomies = {} |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
343 buckets[sn] = source_taxonomies |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
344 for tn in tax_names: |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
345 source_taxonomies[tn] = _TaxonomyTermsInfo() |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
346 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
347 # Now see which ones are 'dirty' based on our bake record. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
348 logger.debug("Gathering dirty taxonomy terms") |
5 | 349 for prev_entry, cur_entry in record.transitions.values(): |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
350 # Re-bake all taxonomy pages that include new or changed |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
351 # pages. |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
352 if cur_entry and cur_entry.was_any_sub_baked: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
353 entries = [cur_entry] |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
354 if prev_entry: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
355 entries.append(prev_entry) |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
356 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
357 for tax in self.app.taxonomies: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
358 changed_terms = set() |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
359 for e in entries: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
360 terms = e.config.get(tax.setting_name) |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
361 if terms: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
362 if not tax.is_multiple: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
363 terms = [terms] |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
364 changed_terms |= set(terms) |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
365 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
366 if len(changed_terms) > 0: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
367 tt_info = buckets[cur_entry.source_name][tax.name] |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
368 tt_info.dirty_terms |= changed_terms |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
369 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
370 # Remember all terms used. |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
371 for tax in self.app.taxonomies: |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
372 if cur_entry and not cur_entry.was_overriden: |
324
65e6d72f3877
bake/serve: Fix how taxonomy index pages are setup and rendered.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
373 cur_terms = cur_entry.config.get(tax.setting_name) |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
374 if cur_terms: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
375 if not tax.is_multiple: |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
376 cur_terms = [cur_terms] |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
377 tt_info = buckets[cur_entry.source_name][tax.name] |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
378 tt_info.all_terms |= set(cur_terms) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
379 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
380 # Re-bake the combination pages for terms that are 'dirty'. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
381 known_combinations = set() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
382 logger.debug("Gathering dirty term combinations") |
5 | 383 for prev_entry, cur_entry in record.transitions.values(): |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
384 if not cur_entry: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
385 continue |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
386 used_taxonomy_terms = cur_entry.getAllUsedTaxonomyTerms() |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
387 for sn, tn, terms in used_taxonomy_terms: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
388 if isinstance(terms, tuple): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
389 known_combinations.add((sn, tn, terms)) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
390 for sn, tn, terms in known_combinations: |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
391 tt_info = buckets[sn][tn] |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
392 tt_info.all_terms.add(terms) |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
393 if not tt_info.dirty_terms.isdisjoint(set(terms)): |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
394 tt_info.dirty_terms.add(terms) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
395 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
396 return buckets |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
397 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
398 def _bakeTaxonomyBuckets(self, record, pool, buckets): |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
399 # Start baking those terms. |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
400 expected_result_count = 0 |
5 | 401 for source_name, source_taxonomies in buckets.items(): |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
402 for tax_name, tt_info in source_taxonomies.items(): |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
403 terms = tt_info.dirty_terms |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
404 if len(terms) == 0: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
405 continue |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
406 |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
407 logger.debug( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
408 "Baking '%s' for source '%s': %s" % |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
409 (tax_name, source_name, terms)) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
410 tax = self.app.getTaxonomy(tax_name) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
411 source = self.app.getSource(source_name) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
412 tax_page_ref = tax.getPageRef(source) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
413 if not tax_page_ref.exists: |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
414 logger.debug( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
415 "No taxonomy page found at '%s', skipping." % |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
416 tax.page_ref) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
417 continue |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
418 |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
419 logger.debug( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
420 "Using taxonomy page: %s:%s" % |
369
4b1019bb2533
serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents:
363
diff
changeset
|
421 (tax_page_ref.source_name, tax_page_ref.rel_path)) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
422 fac = tax_page_ref.getFactory() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
423 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
424 for term in terms: |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
425 logger.debug( |
369
4b1019bb2533
serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents:
363
diff
changeset
|
426 "Queuing: %s [%s=%s]" % |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
427 (fac.ref_spec, tax_name, term)) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
428 tax_info = TaxonomyInfo(tax_name, source_name, term) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
429 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
430 cur_entry = BakeRecordEntry( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
431 fac.source.name, fac.path, tax_info) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
432 record.addEntry(cur_entry) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
433 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
434 if self._queueBakeJob(record, pool, fac, tax_info): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
435 expected_result_count += 1 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
436 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
437 def _handler(res): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
438 entry = record.getCurrentEntry(res.path, res.taxonomy_info) |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
439 entry.subs = res.sub_entries |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
440 if res.errors: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
441 entry.errors += res.errors |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
442 if entry.has_any_error: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
443 record.current.success = False |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
444 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
445 self._waitOnWorkerPool( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
446 pool, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
447 expected_result_count=expected_result_count, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
448 result_handler=_handler) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
449 |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
450 # Now we create bake entries for all the terms that were *not* dirty. |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
451 # This is because otherwise, on the next incremental bake, we wouldn't |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
452 # find any entry for those things, and figure that we need to delete |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
453 # their outputs. |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
454 for prev_entry, cur_entry in record.transitions.values(): |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
455 # Only consider taxonomy-related entries that don't have any |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
456 # current version. |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
457 if (prev_entry and prev_entry.taxonomy_info and |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
458 not cur_entry): |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
459 ti = prev_entry.taxonomy_info |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
460 tt_info = buckets[ti.source_name][ti.taxonomy_name] |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
461 if ti.term in tt_info.all_terms: |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
462 logger.debug("Creating unbaked entry for taxonomy " |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
463 "term '%s:%s'." % (ti.taxonomy_name, ti.term)) |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
464 record.collapseEntry(prev_entry) |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
465 else: |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
466 logger.debug("Taxonomy term '%s:%s' isn't used anymore." % |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
467 (ti.taxonomy_name, ti.term)) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
468 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
469 return expected_result_count |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
470 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
471 def _queueBakeJob(self, record, pool, fac, tax_info=None): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
472 # Get the previous (if any) and current entry for this page. |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
473 pair = record.getPreviousAndCurrentEntries(fac.path, tax_info) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
474 assert pair is not None |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
475 prev_entry, cur_entry = pair |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
476 assert cur_entry is not None |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
477 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
478 # Ignore if there were errors in the previous passes. |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
479 if cur_entry.errors: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
480 logger.debug("Ignoring %s because it had previous " |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
481 "errors." % fac.ref_spec) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
482 return False |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
483 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
484 # Build the route metadata and find the appropriate route. |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
485 route_metadata = copy.deepcopy(fac.metadata) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
486 if tax_info is not None: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
487 tax = self.app.getTaxonomy(tax_info.taxonomy_name) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
488 route = self.app.getTaxonomyRoute(tax_info.taxonomy_name, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
489 tax_info.source_name) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
490 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
491 slugified_term = route.slugifyTaxonomyTerm(tax_info.term) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
492 route_metadata[tax.term_name] = slugified_term |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
493 else: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
494 route = self.app.getRoute(fac.source.name, route_metadata, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
495 skip_taxonomies=True) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
496 assert route is not None |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
497 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
498 # Figure out if this page is overriden by another previously |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
499 # baked page. This happens for example when the user has |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
500 # made a page that has the same page/URL as a theme page. |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
501 page = fac.buildPage() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
502 uri = route.getUri(route_metadata, provider=page) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
503 override_entry = record.getOverrideEntry(page.path, uri) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
504 if override_entry is not None: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
505 override_source = self.app.getSource( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
506 override_entry.source_name) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
507 if override_source.realm == fac.source.realm: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
508 cur_entry.errors.append( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
509 "Page '%s' maps to URL '%s' but is overriden " |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
510 "by page '%s'." % |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
511 (fac.ref_spec, uri, override_entry.path)) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
512 logger.error(cur_entry.errors[-1]) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
513 cur_entry.flags |= BakeRecordEntry.FLAG_OVERRIDEN |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
514 return False |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
515 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
516 job = BakeWorkerJob( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
517 JOB_BAKE, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
518 BakeJobPayload(fac, route_metadata, prev_entry, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
519 record.dirty_source_names, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
520 tax_info)) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
521 pool.queue.put_nowait(job) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
522 return True |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
523 |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
524 def _handleDeletetions(self, record): |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
525 logger.debug("Handling deletions...") |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
526 for path, reason in record.getDeletions(): |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
527 logger.debug("Removing '%s': %s" % (path, reason)) |
146
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
528 try: |
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
529 os.remove(path) |
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
530 logger.info('[delete] %s' % path) |
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
531 except OSError: |
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
532 # Not a big deal if that file had already been removed |
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
533 # by the user. |
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
534 pass |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
535 |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
536 def _logErrors(self, path, errors): |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
537 rel_path = os.path.relpath(path, self.app.root_dir) |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
538 logger.error("Errors found in %s:" % rel_path) |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
539 for e in errors: |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
540 logger.error(" " + e) |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
541 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
542 def _createWorkerPool(self): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
543 from piecrust.baking.worker import BakeWorkerContext, worker_func |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
544 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
545 pool = _WorkerPool() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
546 for i in range(self.num_workers): |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
547 ctx = BakeWorkerContext( |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
548 self.app.root_dir, self.out_dir, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
549 pool.queue, pool.results, pool.abort_event, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
550 force=self.force, debug=self.app.debug) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
551 w = multiprocessing.Process( |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
552 name='Worker_%d' % i, |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
553 target=worker_func, args=(i, ctx)) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
554 w.start() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
555 pool.workers.append(w) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
556 return pool |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
557 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
558 def _terminateWorkerPool(self, pool): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
559 pool.abort_event.set() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
560 for w in pool.workers: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
561 w.join() |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
562 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
563 def _waitOnWorkerPool(self, pool, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
564 expected_result_count=-1, result_handler=None): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
565 assert result_handler is None or expected_result_count >= 0 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
566 abort_with_exception = None |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
567 try: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
568 if result_handler is None: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
569 pool.queue.join() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
570 else: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
571 got_count = 0 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
572 while got_count < expected_result_count: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
573 try: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
574 res = pool.results.get(True, 10) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
575 except queue.Empty: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
576 logger.error( |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
577 "Got %d results, expected %d, and timed-out " |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
578 "for 10 seconds. A worker might be stuck?" % |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
579 (got_count, expected_result_count)) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
580 abort_with_exception = Exception("Worker time-out.") |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
581 break |
159
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
582 |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
583 if isinstance(res, dict) and res.get('type') == 'error': |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
584 abort_with_exception = Exception( |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
585 'Worker critical error:\n' + |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
586 '\n'.join(res['messages'])) |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
587 break |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
588 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
589 got_count += 1 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
590 result_handler(res) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
591 except KeyboardInterrupt as kiex: |
159
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
592 logger.warning("Bake aborted by user... " |
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
593 "waiting for workers to stop.") |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
594 abort_with_exception = kiex |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
595 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
596 if abort_with_exception: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
597 pool.abort_event.set() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
598 for w in pool.workers: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
599 w.join(2) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
600 raise abort_with_exception |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
601 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
602 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
603 class _WorkerPool(object): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
604 def __init__(self): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
605 self.queue = multiprocessing.JoinableQueue() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
606 self.results = multiprocessing.Queue() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
607 self.abort_event = multiprocessing.Event() |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
608 self.workers = [] |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
609 |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
610 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
611 class _TaxonomyTermsInfo(object): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
612 def __init__(self): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
613 self.dirty_terms = set() |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
614 self.all_terms = set() |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
615 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
616 def __str__(self): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
617 return 'dirty:%s, all:%s' % (self.dirty_terms, self.all_terms) |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
618 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
619 def __repr__(self): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
620 return 'dirty:%s, all:%s' % (self.dirty_terms, self.all_terms) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
371
diff
changeset
|
621 |