Mercurial > piecrust2
annotate piecrust/baking/baker.py @ 369:4b1019bb2533
serve: Giant refactor to change how we handle data when serving pages.
* We need a distinction between source metadata and route metadata. In most
cases they're the same, but in cases like taxonomy pages, route metadata
contains more things that can't be in source metadata if we want to re-use
cached pages.
* Create a new `QualifiedPage` type which is a page with a specific route
and route metadata. Pass this around in many places.
* Instead of passing an URL around, use the route in the `QualifiedPage` to
generate URLs. This is better since it removes the guess-work from trying
to generate URLs for sub-pages.
* Deep-copy app and page configurations before passing them around to things
that could modify them, like data builders and such.
* Exclude taxonomy pages from iterator data providers.
* Properly nest iterator data providers for when the theme and user page
sources are merged inside `site.pages`.
author | Ludovic Chabant <ludovic@chabant.com> |
---|---|
date | Sun, 03 May 2015 18:47:10 -0700 |
parents | dd25bd3ce1f9 |
children | c2ca72fb7f0b |
rev | line source |
---|---|
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
1 import time |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
2 import os.path |
25
65ae19c4e8a3
Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents:
23
diff
changeset
|
3 import shutil |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
4 import hashlib |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
5 import logging |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
6 import threading |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
7 from piecrust.baking.records import ( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
8 TransitionalBakeRecord, BakeRecordPageEntry) |
150
91dcbb5fe1e8
Split baking code in smaller files.
Ludovic Chabant <ludovic@chabant.com>
parents:
147
diff
changeset
|
9 from piecrust.baking.scheduler import BakeScheduler |
91dcbb5fe1e8
Split baking code in smaller files.
Ludovic Chabant <ludovic@chabant.com>
parents:
147
diff
changeset
|
10 from piecrust.baking.single import (BakingError, PageBaker) |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
11 from piecrust.chefutil import format_timed, log_friendly_exception |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
12 from piecrust.sources.base import ( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
13 PageFactory, |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
14 REALM_NAMES, REALM_USER, REALM_THEME) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
15 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
16 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
17 logger = logging.getLogger(__name__) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
18 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
19 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
20 class Baker(object): |
205
e725af1d48fb
bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents:
199
diff
changeset
|
21 def __init__(self, app, out_dir, force=False): |
127
bc63dc20baa0
Fix how we pass the out directory to the baking modules.
Ludovic Chabant <ludovic@chabant.com>
parents:
120
diff
changeset
|
22 assert app and out_dir |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
23 self.app = app |
127
bc63dc20baa0
Fix how we pass the out directory to the baking modules.
Ludovic Chabant <ludovic@chabant.com>
parents:
120
diff
changeset
|
24 self.out_dir = out_dir |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
25 self.force = force |
205
e725af1d48fb
bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents:
199
diff
changeset
|
26 self.num_workers = app.config.get('baker/workers', 4) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
27 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
28 # Remember what taxonomy pages we should skip |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
29 # (we'll bake them repeatedly later with each taxonomy term) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
30 self.taxonomy_pages = [] |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
31 logger.debug("Gathering taxonomy page paths:") |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
32 for tax in self.app.taxonomies: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
33 for src in self.app.sources: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
34 path = tax.resolvePagePath(src.name) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
35 if path is not None: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
36 self.taxonomy_pages.append(path) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
37 logger.debug(" - %s" % path) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
38 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
39 def bake(self): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
40 logger.debug(" Bake Output: %s" % self.out_dir) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
41 logger.debug(" Root URL: %s" % self.app.config.get('site/root')) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
42 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
43 # Get into bake mode. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
44 start_time = time.clock() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
45 self.app.config.set('baker/is_baking', True) |
32
43091c9837bf
Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents:
25
diff
changeset
|
46 self.app.env.base_asset_url_format = '%uri%' |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
47 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
48 # Make sure the output directory exists. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
49 if not os.path.isdir(self.out_dir): |
5 | 50 os.makedirs(self.out_dir, 0o755) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
51 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
52 # Load/create the bake record. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
53 record = TransitionalBakeRecord() |
45
efd0d3bacc9e
Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
42
diff
changeset
|
54 record_cache = self.app.cache.getCache('baker') |
333
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
55 record_id = hashlib.md5(self.out_dir.encode('utf8')).hexdigest() |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
56 record_name = record_id + '.record' |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
57 if not self.force and record_cache.has(record_name): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
58 t = time.clock() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
59 record.loadPrevious(record_cache.getCachePath(record_name)) |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
60 logger.debug(format_timed( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
61 t, 'loaded previous bake record', |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
62 colored=False)) |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
63 record.current.success = True |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
64 |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
65 # Figure out if we need to clean the cache because important things |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
66 # have changed. |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
67 self._handleCacheValidity(record) |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
68 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
69 # Gather all sources by realm -- we're going to bake each realm |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
70 # separately so we can handle "overlaying" (i.e. one realm overrides |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
71 # another realm's pages). |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
72 sources_by_realm = {} |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
73 for source in self.app.sources: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
74 srclist = sources_by_realm.setdefault(source.realm, []) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
75 srclist.append(source) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
76 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
77 # Bake the realms. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
78 realm_list = [REALM_USER, REALM_THEME] |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
79 for realm in realm_list: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
80 srclist = sources_by_realm.get(realm) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
81 if srclist is not None: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
82 self._bakeRealm(record, realm, srclist) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
83 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
84 # Bake taxonomies. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
85 self._bakeTaxonomies(record) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
86 |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
87 # Delete files from the output. |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
88 self._handleDeletetions(record) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
89 |
333
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
90 # Backup previous records. |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
91 for i in range(8, -1, -1): |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
92 suffix = '' if i == 0 else '.%d' % i |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
93 record_path = record_cache.getCachePath( |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
94 '%s%s.record' % (record_id, suffix)) |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
95 if os.path.exists(record_path): |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
96 record_path_next = record_cache.getCachePath( |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
97 '%s.%s.record' % (record_id, i + 1)) |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
98 if os.path.exists(record_path_next): |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
99 os.remove(record_path_next) |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
100 os.rename(record_path, record_path_next) |
91b07f9efdc1
bake: Use a rotating bake record.
Ludovic Chabant <ludovic@chabant.com>
parents:
324
diff
changeset
|
101 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
102 # Save the bake record. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
103 t = time.clock() |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
104 record.current.bake_time = time.time() |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
105 record.current.out_dir = self.out_dir |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
106 record.saveCurrent(record_cache.getCachePath(record_name)) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
107 logger.debug(format_timed(t, 'saved bake record', colored=False)) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
108 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
109 # All done. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
110 self.app.config.set('baker/is_baking', False) |
151
fd146f54bdaa
Forgot this wasn't C++.
Ludovic Chabant <ludovic@chabant.com>
parents:
150
diff
changeset
|
111 logger.debug(format_timed(start_time, 'done baking')) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
112 |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
113 return record.detach() |
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
114 |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
115 def _handleCacheValidity(self, record): |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
116 start_time = time.clock() |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
117 |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
118 reason = None |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
119 if self.force: |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
120 reason = "ordered to" |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
121 elif not self.app.config.get('__cache_valid'): |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
122 # The configuration file was changed, or we're running a new |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
123 # version of the app. |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
124 reason = "not valid anymore" |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
125 elif (not record.previous.bake_time or |
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
126 not record.previous.hasLatestVersion()): |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
127 # We have no valid previous bake record. |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
128 reason = "need bake record regeneration" |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
129 else: |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
130 # Check if any template has changed since the last bake. Since |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
131 # there could be some advanced conditional logic going on, we'd |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
132 # better just force a bake from scratch if that's the case. |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
133 max_time = 0 |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
134 for d in self.app.templates_dirs: |
42
9e058e221108
Fix a crash when checking for timestamps on template files.
Ludovic Chabant <ludovic@chabant.com>
parents:
39
diff
changeset
|
135 for dpath, _, filenames in os.walk(d): |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
136 for fn in filenames: |
42
9e058e221108
Fix a crash when checking for timestamps on template files.
Ludovic Chabant <ludovic@chabant.com>
parents:
39
diff
changeset
|
137 full_fn = os.path.join(dpath, fn) |
9e058e221108
Fix a crash when checking for timestamps on template files.
Ludovic Chabant <ludovic@chabant.com>
parents:
39
diff
changeset
|
138 max_time = max(max_time, os.path.getmtime(full_fn)) |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
139 if max_time >= record.previous.bake_time: |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
140 reason = "templates modified" |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
141 |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
142 if reason is not None: |
85
3471ffa059b2
Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
50
diff
changeset
|
143 # We have to bake everything from scratch. |
105
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
144 for cache_name in self.app.cache.getCacheNames( |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
145 except_names=['app']): |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
146 cache_dir = self.app.cache.getCacheDir(cache_name) |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
147 if os.path.isdir(cache_dir): |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
148 logger.debug("Cleaning baker cache: %s" % cache_dir) |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
149 shutil.rmtree(cache_dir) |
45
efd0d3bacc9e
Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
42
diff
changeset
|
150 self.force = True |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
151 record.incremental_count = 0 |
96
0445a2232de7
Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
91
diff
changeset
|
152 record.clearPrevious() |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
153 logger.info(format_timed( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
154 start_time, |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
155 "cleaned cache (reason: %s)" % reason)) |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
156 else: |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
157 record.incremental_count += 1 |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
158 logger.debug(format_timed( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
159 start_time, "cache is assumed valid", |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
160 colored=False)) |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
161 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
162 def _bakeRealm(self, record, realm, srclist): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
163 # Gather all page factories from the sources and queue them |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
164 # for the workers to pick up. Just skip taxonomy pages for now. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
165 logger.debug("Baking realm %s" % REALM_NAMES[realm]) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
166 pool, queue, abort = self._createWorkerPool(record, self.num_workers) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
167 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
168 for source in srclist: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
169 factories = source.getPageFactories() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
170 for fac in factories: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
171 if fac.path in self.taxonomy_pages: |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
172 logger.debug( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
173 "Skipping taxonomy page: %s:%s" % |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
174 (source.name, fac.ref_spec)) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
175 continue |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
176 |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
177 entry = BakeRecordPageEntry(fac.source.name, fac.rel_path, |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
178 fac.path) |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
179 record.addEntry(entry) |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
180 |
363
dd25bd3ce1f9
serve: Refactoring and fixes to be able to serve taxonomy pages.
Ludovic Chabant <ludovic@chabant.com>
parents:
338
diff
changeset
|
181 route = self.app.getRoute(source.name, fac.metadata, |
dd25bd3ce1f9
serve: Refactoring and fixes to be able to serve taxonomy pages.
Ludovic Chabant <ludovic@chabant.com>
parents:
338
diff
changeset
|
182 skip_taxonomies=True) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
183 if route is None: |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
184 entry.errors.append( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
185 "Can't get route for page: %s" % fac.ref_spec) |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
186 logger.error(entry.errors[-1]) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
187 continue |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
188 |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
189 queue.addJob(BakeWorkerJob(fac, route, entry)) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
190 |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
191 success = self._waitOnWorkerPool(pool, abort) |
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
192 record.current.success &= success |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
193 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
194 def _bakeTaxonomies(self, record): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
195 logger.debug("Baking taxonomies") |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
196 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
197 # Let's see all the taxonomy terms for which we must bake a |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
198 # listing page... first, pre-populate our big map of used terms. |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
199 # For each source name, we have a list of taxonomies, and for each |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
200 # taxonomies, a list of terms, some being 'dirty', some used last |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
201 # time, etc. |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
202 buckets = {} |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
203 tax_names = [t.name for t in self.app.taxonomies] |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
204 source_names = [s.name for s in self.app.sources] |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
205 for sn in source_names: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
206 source_taxonomies = {} |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
207 buckets[sn] = source_taxonomies |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
208 for tn in tax_names: |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
209 source_taxonomies[tn] = _TaxonomyTermsInfo() |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
210 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
211 # Now see which ones are 'dirty' based on our bake record. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
212 logger.debug("Gathering dirty taxonomy terms") |
5 | 213 for prev_entry, cur_entry in record.transitions.values(): |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
214 # Re-bake all taxonomy pages that include new or changed |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
215 # pages. |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
216 if cur_entry and cur_entry.was_any_sub_baked: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
217 entries = [cur_entry] |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
218 if prev_entry: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
219 entries.append(prev_entry) |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
220 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
221 for tax in self.app.taxonomies: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
222 changed_terms = set() |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
223 for e in entries: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
224 terms = e.config.get(tax.setting_name) |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
225 if terms: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
226 if not tax.is_multiple: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
227 terms = [terms] |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
228 changed_terms |= set(terms) |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
229 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
230 if len(changed_terms) > 0: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
231 tt_info = buckets[cur_entry.source_name][tax.name] |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
232 tt_info.dirty_terms |= changed_terms |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
233 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
234 # Remember all terms used. |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
235 for tax in self.app.taxonomies: |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
236 if cur_entry and not cur_entry.was_overriden: |
324
65e6d72f3877
bake/serve: Fix how taxonomy index pages are setup and rendered.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
237 cur_terms = cur_entry.config.get(tax.setting_name) |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
238 if cur_terms: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
239 if not tax.is_multiple: |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
240 cur_terms = [cur_terms] |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
241 tt_info = buckets[cur_entry.source_name][tax.name] |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
242 tt_info.all_terms |= set(cur_terms) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
243 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
244 # Re-bake the combination pages for terms that are 'dirty'. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
245 known_combinations = set() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
246 logger.debug("Gathering dirty term combinations") |
5 | 247 for prev_entry, cur_entry in record.transitions.values(): |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
248 if not cur_entry: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
249 continue |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
250 used_taxonomy_terms = cur_entry.getAllUsedTaxonomyTerms() |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
251 for sn, tn, terms in used_taxonomy_terms: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
252 if isinstance(terms, tuple): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
253 known_combinations.add((sn, tn, terms)) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
254 for sn, tn, terms in known_combinations: |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
255 tt_info = buckets[sn][tn] |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
256 tt_info.all_terms.add(terms) |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
257 if not tt_info.dirty_terms.isdisjoint(set(terms)): |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
258 tt_info.dirty_terms.add(terms) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
259 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
260 # Start baking those terms. |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
261 pool, queue, abort = self._createWorkerPool(record, self.num_workers) |
5 | 262 for source_name, source_taxonomies in buckets.items(): |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
263 for tax_name, tt_info in source_taxonomies.items(): |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
264 terms = tt_info.dirty_terms |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
265 if len(terms) == 0: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
266 continue |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
267 |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
268 logger.debug( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
269 "Baking '%s' for source '%s': %s" % |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
270 (tax_name, source_name, terms)) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
271 tax = self.app.getTaxonomy(tax_name) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
272 route = self.app.getTaxonomyRoute(tax_name, source_name) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
273 tax_page_ref = tax.getPageRef(source_name) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
274 if not tax_page_ref.exists: |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
275 logger.debug( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
276 "No taxonomy page found at '%s', skipping." % |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
277 tax.page_ref) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
278 continue |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
279 |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
280 logger.debug( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
281 "Using taxonomy page: %s:%s" % |
369
4b1019bb2533
serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents:
363
diff
changeset
|
282 (tax_page_ref.source_name, tax_page_ref.rel_path)) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
283 for term in terms: |
369
4b1019bb2533
serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents:
363
diff
changeset
|
284 fac = tax_page_ref.getFactory() |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
285 logger.debug( |
369
4b1019bb2533
serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents:
363
diff
changeset
|
286 "Queuing: %s [%s=%s]" % |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
287 (fac.ref_spec, tax_name, term)) |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
288 entry = BakeRecordPageEntry( |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
289 fac.source.name, fac.rel_path, fac.path, |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
290 (tax_name, term, source_name)) |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
291 record.addEntry(entry) |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
292 queue.addJob(BakeWorkerJob(fac, route, entry)) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
293 |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
294 success = self._waitOnWorkerPool(pool, abort) |
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
295 record.current.success &= success |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
296 |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
297 # Now we create bake entries for all the terms that were *not* dirty. |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
298 # This is because otherwise, on the next incremental bake, we wouldn't |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
299 # find any entry for those things, and figure that we need to delete |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
300 # their outputs. |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
301 for prev_entry, cur_entry in record.transitions.values(): |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
302 # Only consider taxonomy-related entries that don't have any |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
303 # current version. |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
304 if (prev_entry and prev_entry.taxonomy_info and |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
305 not cur_entry): |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
306 sn = prev_entry.source_name |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
307 tn, tt, tsn = prev_entry.taxonomy_info |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
308 tt_info = buckets[tsn][tn] |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
309 if tt in tt_info.all_terms: |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
310 logger.debug("Creating unbaked entry for taxonomy " |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
311 "term '%s:%s'." % (tn, tt)) |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
312 record.collapseEntry(prev_entry) |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
313 else: |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
314 logger.debug("Taxonomy term '%s:%s' isn't used anymore." % |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
315 (tn, tt)) |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
316 |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
317 def _handleDeletetions(self, record): |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
318 for path, reason in record.getDeletions(): |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
319 logger.debug("Removing '%s': %s" % (path, reason)) |
146
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
320 try: |
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
321 os.remove(path) |
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
322 logger.info('[delete] %s' % path) |
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
323 except OSError: |
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
324 # Not a big deal if that file had already been removed |
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
325 # by the user. |
0609739169bd
Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents:
133
diff
changeset
|
326 pass |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
327 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
328 def _createWorkerPool(self, record, pool_size=4): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
329 pool = [] |
85
3471ffa059b2
Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
50
diff
changeset
|
330 queue = BakeScheduler(record) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
331 abort = threading.Event() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
332 for i in range(pool_size): |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
333 ctx = BakeWorkerContext( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
334 self.app, self.out_dir, self.force, |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
335 record, queue, abort) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
336 worker = BakeWorker(i, ctx) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
337 pool.append(worker) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
338 return pool, queue, abort |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
339 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
340 def _waitOnWorkerPool(self, pool, abort): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
341 for w in pool: |
85
3471ffa059b2
Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
50
diff
changeset
|
342 w.start() |
159
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
343 |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
344 success = True |
159
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
345 try: |
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
346 for w in pool: |
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
347 w.join() |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
348 success &= w.success |
159
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
349 except KeyboardInterrupt: |
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
350 logger.warning("Bake aborted by user... " |
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
351 "waiting for workers to stop.") |
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
352 abort.set() |
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
353 for w in pool: |
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
354 w.join() |
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
355 raise |
232989a6df36
Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents:
158
diff
changeset
|
356 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
357 if abort.is_set(): |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
358 excs = [w.abort_exception for w in pool |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
359 if w.abort_exception is not None] |
85
3471ffa059b2
Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
50
diff
changeset
|
360 logger.error("Baking was aborted due to %s error(s):" % len(excs)) |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
361 if self.app.debug: |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
362 for e in excs: |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
363 logger.exception(e) |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
364 else: |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
365 for e in excs: |
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
366 log_friendly_exception(logger, e) |
105
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
367 raise BakingError("Baking was aborted due to errors.") |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
368 |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
369 return success |
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
370 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
371 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
372 class BakeWorkerContext(object): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
373 def __init__(self, app, out_dir, force, record, work_queue, |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
374 abort_event): |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
375 self.app = app |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
376 self.out_dir = out_dir |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
377 self.force = force |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
378 self.record = record |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
379 self.work_queue = work_queue |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
380 self.abort_event = abort_event |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
381 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
382 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
383 class BakeWorkerJob(object): |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
384 def __init__(self, factory, route, record_entry): |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
385 self.factory = factory |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
386 self.route = route |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
387 self.record_entry = record_entry |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
388 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
389 @property |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
390 def source(self): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
391 return self.factory.source |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
392 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
393 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
394 class BakeWorker(threading.Thread): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
395 def __init__(self, wid, ctx): |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
396 super(BakeWorker, self).__init__(name=('worker%d' % wid)) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
397 self.wid = wid |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
398 self.ctx = ctx |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
399 self.abort_exception = None |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
400 self.success = True |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
401 self._page_baker = PageBaker( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
402 ctx.app, ctx.out_dir, ctx.force, |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
403 ctx.record) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
404 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
405 def run(self): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
406 while(not self.ctx.abort_event.is_set()): |
96
0445a2232de7
Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
91
diff
changeset
|
407 try: |
0445a2232de7
Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
91
diff
changeset
|
408 job = self.ctx.work_queue.getNextJob(wait_timeout=1) |
0445a2232de7
Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
91
diff
changeset
|
409 if job is None: |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
410 logger.debug( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
411 "[%d] No more work... shutting down." % |
96
0445a2232de7
Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
91
diff
changeset
|
412 self.wid) |
0445a2232de7
Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
91
diff
changeset
|
413 break |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
414 success = self._unsafeRun(job) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
415 logger.debug("[%d] Done with page." % self.wid) |
85
3471ffa059b2
Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
50
diff
changeset
|
416 self.ctx.work_queue.onJobFinished(job) |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
417 self.success &= success |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
418 except Exception as ex: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
419 self.ctx.abort_event.set() |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
420 self.abort_exception = ex |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
421 self.success = False |
39
2f717f961996
Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents:
36
diff
changeset
|
422 logger.debug("[%d] Critical error, aborting." % self.wid) |
50
2fec3ee1298f
Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents:
49
diff
changeset
|
423 if self.ctx.app.debug: |
2fec3ee1298f
Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents:
49
diff
changeset
|
424 logger.exception(ex) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
425 break |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
426 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
427 def _unsafeRun(self, job): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
428 start_time = time.clock() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
429 |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
430 entry = job.record_entry |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
431 try: |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
333
diff
changeset
|
432 self._page_baker.bake(job.factory, job.route, entry) |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
433 except BakingError as ex: |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
434 logger.debug("Got baking error. Adding it to the record.") |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
435 while ex: |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
436 entry.errors.append(str(ex)) |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
437 ex = ex.__cause__ |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
438 |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
439 has_error = False |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
440 for e in entry.getAllErrors(): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
441 has_error = True |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
442 logger.error(e) |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
443 if has_error: |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
444 return False |
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
445 |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
446 if entry.was_any_sub_baked: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
447 first_sub = entry.subs[0] |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
448 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
449 friendly_uri = first_sub.out_uri |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
450 if friendly_uri == '': |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
451 friendly_uri = '[main page]' |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
452 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
453 friendly_count = '' |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
454 if entry.num_subs > 1: |
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
455 friendly_count = ' (%d pages)' % entry.num_subs |
158
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
456 logger.info(format_timed( |
1187739e5a19
Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents:
151
diff
changeset
|
457 start_time, '[%d] %s%s' % |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
458 (self.wid, friendly_uri, friendly_count))) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
459 |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
460 return True |
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
205
diff
changeset
|
461 |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
462 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
463 class _TaxonomyTermsInfo(object): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
464 def __init__(self): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
465 self.dirty_terms = set() |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
466 self.all_terms = set() |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
467 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
468 def __str__(self): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
469 return 'dirty:%s, all:%s' % (self.dirty_terms, self.all_terms) |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
470 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
471 def __repr__(self): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
472 return 'dirty:%s, all:%s' % (self.dirty_terms, self.all_terms) |