Mercurial > piecrust2

annotate piecrust/baking/baker.py @ 217:1f4c3dae1fe8

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Better error handling for site baking. The site baker now keeps track of whether any worker saw any error. It returns the current record after a bake. The `bake` command uses this return value to figure out what kind of exit code to return.
author Ludovic Chabant <ludovic@chabant.com>
date Sat, 31 Jan 2015 17:35:36 -0800
parents e725af1d48fb
children 65e6d72f3877
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import time
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
25
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 23
diff changeset
3 import shutil
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import hashlib
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import logging
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 import threading
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
7 from piecrust.baking.records import (
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
8 TransitionalBakeRecord, BakeRecordPageEntry)
150
91dcbb5fe1e8 Split baking code in smaller files.
Ludovic Chabant <ludovic@chabant.com>
parents: 147
diff changeset
9 from piecrust.baking.scheduler import BakeScheduler
91dcbb5fe1e8 Split baking code in smaller files.
Ludovic Chabant <ludovic@chabant.com>
parents: 147
diff changeset
10 from piecrust.baking.single import (BakingError, PageBaker)
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
11 from piecrust.chefutil import format_timed, log_friendly_exception
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
12 from piecrust.sources.base import (
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
13 PageFactory,
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 REALM_NAMES, REALM_USER, REALM_THEME)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 class Baker(object):
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 199
diff changeset
21 def __init__(self, app, out_dir, force=False):
127
bc63dc20baa0 Fix how we pass the out directory to the baking modules.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
22 assert app and out_dir
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 self.app = app
127
bc63dc20baa0 Fix how we pass the out directory to the baking modules.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
24 self.out_dir = out_dir
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 self.force = force
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 199
diff changeset
26 self.num_workers = app.config.get('baker/workers', 4)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 # Remember what taxonomy pages we should skip
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29 # (we'll bake them repeatedly later with each taxonomy term)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 self.taxonomy_pages = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 logger.debug("Gathering taxonomy page paths:")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 for tax in self.app.taxonomies:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 for src in self.app.sources:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 path = tax.resolvePagePath(src.name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35 if path is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 self.taxonomy_pages.append(path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 logger.debug(" - %s" % path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 def bake(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 logger.debug(" Bake Output: %s" % self.out_dir)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 logger.debug(" Root URL: %s" % self.app.config.get('site/root'))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 # Get into bake mode.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 start_time = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 self.app.config.set('baker/is_baking', True)
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 25
diff changeset
46 self.app.env.base_asset_url_format = '%uri%'
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 # Make sure the output directory exists.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 if not os.path.isdir(self.out_dir):
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
50 os.makedirs(self.out_dir, 0o755)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 # Load/create the bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 record = TransitionalBakeRecord()
45
efd0d3bacc9e Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents: 42
diff changeset
54 record_cache = self.app.cache.getCache('baker')
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
55 record_name = (
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
56 hashlib.md5(self.out_dir.encode('utf8')).hexdigest() +
7
343d08ef5668 More PieCrust 3 fixes, and a couple of miscellaneous bug fixes.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
57 '.record')
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 if not self.force and record_cache.has(record_name):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 t = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60 record.loadPrevious(record_cache.getCachePath(record_name))
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
61 logger.debug(format_timed(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
62 t, 'loaded previous bake record',
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
63 colored=False))
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
64 record.current.success = True
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
66 # Figure out if we need to clean the cache because important things
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
67 # have changed.
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
68 self._handleCacheValidity(record)
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
69
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70 # Gather all sources by realm -- we're going to bake each realm
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 # separately so we can handle "overlaying" (i.e. one realm overrides
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 # another realm's pages).
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 sources_by_realm = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 for source in self.app.sources:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 srclist = sources_by_realm.setdefault(source.realm, [])
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 srclist.append(source)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 # Bake the realms.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 realm_list = [REALM_USER, REALM_THEME]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 for realm in realm_list:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81 srclist = sources_by_realm.get(realm)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82 if srclist is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83 self._bakeRealm(record, realm, srclist)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
85 # Bake taxonomies.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
86 self._bakeTaxonomies(record)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
87
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
88 # Delete files from the output.
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
89 self._handleDeletetions(record)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
90
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
91 # Save the bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
92 t = time.clock()
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
93 record.current.bake_time = time.time()
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
94 record.current.out_dir = self.out_dir
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
95 record.collapseRecords()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
96 record.saveCurrent(record_cache.getCachePath(record_name))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
97 logger.debug(format_timed(t, 'saved bake record', colored=False))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
99 # All done.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
100 self.app.config.set('baker/is_baking', False)
151
fd146f54bdaa Forgot this wasn't C++.
Ludovic Chabant <ludovic@chabant.com>
parents: 150
diff changeset
101 logger.debug(format_timed(start_time, 'done baking'))
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
102
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
103 return record.detach()
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
104
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
105 def _handleCacheValidity(self, record):
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
106 start_time = time.clock()
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
107
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
108 reason = None
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
109 if self.force:
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
110 reason = "ordered to"
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
111 elif not self.app.config.get('__cache_valid'):
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
112 # The configuration file was changed, or we're running a new
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
113 # version of the app.
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
114 reason = "not valid anymore"
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
115 elif (not record.previous.bake_time or
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
116 not record.previous.hasLatestVersion()):
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
117 # We have no valid previous bake record.
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
118 reason = "need bake record regeneration"
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
119 else:
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
120 # Check if any template has changed since the last bake. Since
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
121 # there could be some advanced conditional logic going on, we'd
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
122 # better just force a bake from scratch if that's the case.
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
123 max_time = 0
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
124 for d in self.app.templates_dirs:
42
9e058e221108 Fix a crash when checking for timestamps on template files.
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
125 for dpath, _, filenames in os.walk(d):
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
126 for fn in filenames:
42
9e058e221108 Fix a crash when checking for timestamps on template files.
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
127 full_fn = os.path.join(dpath, fn)
9e058e221108 Fix a crash when checking for timestamps on template files.
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
128 max_time = max(max_time, os.path.getmtime(full_fn))
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
129 if max_time >= record.previous.bake_time:
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
130 reason = "templates modified"
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
131
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
132 if reason is not None:
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 50
diff changeset
133 # We have to bake everything from scratch.
105
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
134 for cache_name in self.app.cache.getCacheNames(
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
135 except_names=['app']):
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
136 cache_dir = self.app.cache.getCacheDir(cache_name)
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
137 if os.path.isdir(cache_dir):
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
138 logger.debug("Cleaning baker cache: %s" % cache_dir)
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
139 shutil.rmtree(cache_dir)
45
efd0d3bacc9e Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents: 42
diff changeset
140 self.force = True
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
141 record.incremental_count = 0
96
0445a2232de7 Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 91
diff changeset
142 record.clearPrevious()
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
143 logger.info(format_timed(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
144 start_time,
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
145 "cleaned cache (reason: %s)" % reason))
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
146 else:
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
147 record.incremental_count += 1
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
148 logger.debug(format_timed(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
149 start_time, "cache is assumed valid",
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
150 colored=False))
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
151
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
152 def _bakeRealm(self, record, realm, srclist):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
153 # Gather all page factories from the sources and queue them
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
154 # for the workers to pick up. Just skip taxonomy pages for now.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
155 logger.debug("Baking realm %s" % REALM_NAMES[realm])
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
156 pool, queue, abort = self._createWorkerPool(record, self.num_workers)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
157
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
158 for source in srclist:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
159 factories = source.getPageFactories()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
160 for fac in factories:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
161 if fac.path in self.taxonomy_pages:
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
162 logger.debug(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
163 "Skipping taxonomy page: %s:%s" %
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
164 (source.name, fac.ref_spec))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
165 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
166
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
167 entry = BakeRecordPageEntry(fac)
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
168 record.addEntry(entry)
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
169
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
170 route = self.app.getRoute(source.name, fac.metadata)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
171 if route is None:
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
172 entry.errors.append(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
173 "Can't get route for page: %s" % fac.ref_spec)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
174 logger.error(entry.errors[-1])
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
175 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
176
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
177 queue.addJob(BakeWorkerJob(fac, route, entry))
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
178
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
179 success = self._waitOnWorkerPool(pool, abort)
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
180 record.current.success &= success
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
181
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
182 def _bakeTaxonomies(self, record):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
183 logger.debug("Baking taxonomies")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
184
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
185 # Let's see all the taxonomy terms for which we must bake a
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
186 # listing page... first, pre-populate our big map of used terms.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
187 buckets = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
188 tax_names = [t.name for t in self.app.taxonomies]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
189 source_names = [s.name for s in self.app.sources]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
190 for sn in source_names:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
191 source_taxonomies = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
192 buckets[sn] = source_taxonomies
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
193 for tn in tax_names:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
194 source_taxonomies[tn] = set()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
195
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
196 # Now see which ones are 'dirty' based on our bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
197 logger.debug("Gathering dirty taxonomy terms")
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
198 for prev_entry, cur_entry in record.transitions.values():
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
199 for tax in self.app.taxonomies:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
200 changed_terms = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
201 # Re-bake all taxonomy pages that include new or changed
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
202 # pages.
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
203 if (not prev_entry and cur_entry and
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
204 cur_entry.was_baked_successfully):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
205 changed_terms = cur_entry.config.get(tax.name)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
206 elif (prev_entry and cur_entry and
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
207 cur_entry.was_baked_successfully):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
208 changed_terms = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
209 prev_terms = prev_entry.config.get(tax.name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
210 cur_terms = cur_entry.config.get(tax.name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
211 if tax.is_multiple:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
212 if prev_terms is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
213 changed_terms += prev_terms
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
214 if cur_terms is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
215 changed_terms += cur_terms
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
216 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
217 if prev_terms is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
218 changed_terms.append(prev_terms)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
219 if cur_terms is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
220 changed_terms.append(cur_terms)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
221 if changed_terms is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
222 if not isinstance(changed_terms, list):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
223 changed_terms = [changed_terms]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
224 buckets[cur_entry.source_name][tax.name] |= (
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
225 set(changed_terms))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
226
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
227 # Re-bake the combination pages for terms that are 'dirty'.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
228 known_combinations = set()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
229 logger.debug("Gathering dirty term combinations")
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
230 for prev_entry, cur_entry in record.transitions.values():
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
231 if cur_entry:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
232 known_combinations |= cur_entry.used_taxonomy_terms
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
233 elif prev_entry:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
234 known_combinations |= prev_entry.used_taxonomy_terms
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
235 for sn, tn, terms in known_combinations:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
236 changed_terms = buckets[sn][tn]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
237 if not changed_terms.isdisjoint(set(terms)):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
238 changed_terms.add(terms)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
239
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
240 # Start baking those terms.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
241 pool, queue, abort = self._createWorkerPool(record, self.num_workers)
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
242 for source_name, source_taxonomies in buckets.items():
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
243 for tax_name, terms in source_taxonomies.items():
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
244 if len(terms) == 0:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
245 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
246
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
247 logger.debug(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
248 "Baking '%s' for source '%s': %s" %
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
249 (tax_name, source_name, terms))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
250 tax = self.app.getTaxonomy(tax_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
251 route = self.app.getTaxonomyRoute(tax_name, source_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
252 tax_page_ref = tax.getPageRef(source_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
253 if not tax_page_ref.exists:
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
254 logger.debug(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
255 "No taxonomy page found at '%s', skipping." %
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
256 tax.page_ref)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
257 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
258
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
259 tax_page_source = tax_page_ref.source
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
260 tax_page_rel_path = tax_page_ref.rel_path
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
261 logger.debug(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
262 "Using taxonomy page: %s:%s" %
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
263 (tax_page_source.name, tax_page_rel_path))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
264
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
265 for term in terms:
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
266 fac = PageFactory(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
267 tax_page_source, tax_page_rel_path,
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
268 {tax.term_name: term})
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
269 logger.debug(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
270 "Queuing: %s [%s, %s]" %
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
271 (fac.ref_spec, tax_name, term))
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
272 entry = BakeRecordPageEntry(fac, tax_name, term)
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
273 record.addEntry(entry)
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 50
diff changeset
274 queue.addJob(
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
275 BakeWorkerJob(fac, route, entry, tax_name, term))
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
276
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
277 success = self._waitOnWorkerPool(pool, abort)
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
278 record.current.success &= success
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
279
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
280 def _handleDeletetions(self, record):
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
281 for path, reason in record.getDeletions():
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
282 logger.debug("Removing '%s': %s" % (path, reason))
146
0609739169bd Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
283 try:
0609739169bd Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
284 os.remove(path)
0609739169bd Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
285 logger.info('[delete] %s' % path)
0609739169bd Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
286 except OSError:
0609739169bd Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
287 # Not a big deal if that file had already been removed
0609739169bd Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
288 # by the user.
0609739169bd Don't fail if trying to clean up a file that has already been deleted.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
289 pass
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
290
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
291 def _createWorkerPool(self, record, pool_size=4):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
292 pool = []
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 50
diff changeset
293 queue = BakeScheduler(record)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
294 abort = threading.Event()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
295 for i in range(pool_size):
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
296 ctx = BakeWorkerContext(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
297 self.app, self.out_dir, self.force,
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
298 record, queue, abort)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
299 worker = BakeWorker(i, ctx)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
300 pool.append(worker)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
301 return pool, queue, abort
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
302
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
303 def _waitOnWorkerPool(self, pool, abort):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
304 for w in pool:
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 50
diff changeset
305 w.start()
159
232989a6df36 Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents: 158
diff changeset
306
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
307 success = True
159
232989a6df36 Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents: 158
diff changeset
308 try:
232989a6df36 Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents: 158
diff changeset
309 for w in pool:
232989a6df36 Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents: 158
diff changeset
310 w.join()
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
311 success &= w.success
159
232989a6df36 Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents: 158
diff changeset
312 except KeyboardInterrupt:
232989a6df36 Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents: 158
diff changeset
313 logger.warning("Bake aborted by user... "
232989a6df36 Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents: 158
diff changeset
314 "waiting for workers to stop.")
232989a6df36 Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents: 158
diff changeset
315 abort.set()
232989a6df36 Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents: 158
diff changeset
316 for w in pool:
232989a6df36 Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents: 158
diff changeset
317 w.join()
232989a6df36 Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents: 158
diff changeset
318 raise
232989a6df36 Add support for KeyboardInterrupt in bake process.
Ludovic Chabant <ludovic@chabant.com>
parents: 158
diff changeset
319
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
320 if abort.is_set():
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
321 excs = [w.abort_exception for w in pool
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
322 if w.abort_exception is not None]
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 50
diff changeset
323 logger.error("Baking was aborted due to %s error(s):" % len(excs))
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
324 if self.app.debug:
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
325 for e in excs:
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
326 logger.exception(e)
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
327 else:
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
328 for e in excs:
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
329 log_friendly_exception(logger, e)
105
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
330 raise BakingError("Baking was aborted due to errors.")
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
331
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
332 return success
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
333
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
334
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
335 class BakeWorkerContext(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
336 def __init__(self, app, out_dir, force, record, work_queue,
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
337 abort_event):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
338 self.app = app
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
339 self.out_dir = out_dir
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
340 self.force = force
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
341 self.record = record
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
342 self.work_queue = work_queue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
343 self.abort_event = abort_event
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
344
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
345
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
346 class BakeWorkerJob(object):
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
347 def __init__(self, factory, route, record_entry,
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
348 taxonomy_name=None, taxonomy_term=None):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
349 self.factory = factory
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
350 self.route = route
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
351 self.record_entry = record_entry
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
352 self.taxonomy_name = taxonomy_name
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
353 self.taxonomy_term = taxonomy_term
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
354
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
355 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
356 def source(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
357 return self.factory.source
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
358
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
359
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
360 class BakeWorker(threading.Thread):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
361 def __init__(self, wid, ctx):
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
362 super(BakeWorker, self).__init__(name=('worker%d' % wid))
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
363 self.wid = wid
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
364 self.ctx = ctx
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
365 self.abort_exception = None
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
366 self.success = True
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
367 self._page_baker = PageBaker(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
368 ctx.app, ctx.out_dir, ctx.force,
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
369 ctx.record)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
370
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
371 def run(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
372 while(not self.ctx.abort_event.is_set()):
96
0445a2232de7 Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 91
diff changeset
373 try:
0445a2232de7 Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 91
diff changeset
374 job = self.ctx.work_queue.getNextJob(wait_timeout=1)
0445a2232de7 Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 91
diff changeset
375 if job is None:
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
376 logger.debug(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
377 "[%d] No more work... shutting down." %
96
0445a2232de7 Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 91
diff changeset
378 self.wid)
0445a2232de7 Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 91
diff changeset
379 break
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
380 success = self._unsafeRun(job)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
381 logger.debug("[%d] Done with page." % self.wid)
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 50
diff changeset
382 self.ctx.work_queue.onJobFinished(job)
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
383 self.success &= success
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
384 except Exception as ex:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
385 self.ctx.abort_event.set()
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
386 self.abort_exception = ex
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
387 self.success = False
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
388 logger.debug("[%d] Critical error, aborting." % self.wid)
50
2fec3ee1298f Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents: 49
diff changeset
389 if self.ctx.app.debug:
2fec3ee1298f Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents: 49
diff changeset
390 logger.exception(ex)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
391 break
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
392
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
393 def _unsafeRun(self, job):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
394 start_time = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
395
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
396 entry = job.record_entry
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
397 try:
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
398 self._page_baker.bake(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
399 job.factory, job.route, entry,
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
400 taxonomy_name=job.taxonomy_name,
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
401 taxonomy_term=job.taxonomy_term)
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
402 except BakingError as ex:
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
403 logger.debug("Got baking error. Adding it to the record.")
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
404 while ex:
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
405 entry.errors.append(str(ex))
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
406 ex = ex.__cause__
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
407
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
408 if entry.errors:
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
409 for e in entry.errors:
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
410 logger.error(e)
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
411 return False
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
412
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 105
diff changeset
413 if entry.was_baked_successfully:
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
414 uri = entry.out_uris[0]
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
415 friendly_uri = uri if uri != '' else '[main page]'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
416 friendly_count = ''
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
417 if entry.num_subs > 1:
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
418 friendly_count = ' (%d pages)' % entry.num_subs
158
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
419 logger.info(format_timed(
1187739e5a19 Fix some indentation and line lengths.
Ludovic Chabant <ludovic@chabant.com>
parents: 151
diff changeset
420 start_time, '[%d] %s%s' %
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
421 (self.wid, friendly_uri, friendly_count)))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
422
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
423 return True
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
424