piecrust2: piecrust/baking/baker.py annotate

annotate piecrust/baking/baker.py @ 334:b034f6f15e22

bake: Several bug taxonomy-related fixes for incorrect incremental bakes. * Improve how the baker processes taxonomy terms and figures out what needs to be re-baked or not. * Create bake entries for clean taxnomy terms so they're not deleted by an incremental bake. * Add more information to bake records. * Slugify taxonomy terms is now done by the route in one place. * Fix a bug where the cache key for invalidating rendered segments was not computed the same way as when the caching was done. * Fix how term combinations are passed around, rendered, printed, parsed, etc. (TODO: more word needed in the routing functions) * Expose to the template whether a taxonomy term is a combination or not. * Display term combinations better in the built-in theme. * Rename `route.taxonomy` to `route.taxonomy_name` to prevent confusion. * Add options to show bake records for previous bakes.

author	Ludovic Chabant <ludovic@chabant.com>
date	Fri, 03 Apr 2015 10:59:50 -0700
parents	91b07f9efdc1
children	938be93215cb

rev	line source
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import time
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os.path
25 65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	3 import shutil
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import hashlib
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import logging
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 import threading
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	7 from piecrust.baking.records import (
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	8 TransitionalBakeRecord, BakeRecordPageEntry)
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: 147 diff changeset	9 from piecrust.baking.scheduler import BakeScheduler
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: 147 diff changeset	10 from piecrust.baking.single import (BakingError, PageBaker)
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	11 from piecrust.chefutil import format_timed, log_friendly_exception
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	12 from piecrust.sources.base import (
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	13 PageFactory,
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	14 REALM_NAMES, REALM_USER, REALM_THEME)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	15
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20 class Baker(object):
205 e725af1d48fb bake: Changes in how assets directories are configured. Ludovic Chabant <ludovic@chabant.com> parents: 199 diff changeset	21 def __init__(self, app, out_dir, force=False):
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	22 assert app and out_dir
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	23 self.app = app
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	24 self.out_dir = out_dir
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	25 self.force = force
205 e725af1d48fb bake: Changes in how assets directories are configured. Ludovic Chabant <ludovic@chabant.com> parents: 199 diff changeset	26 self.num_workers = app.config.get('baker/workers', 4)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 # Remember what taxonomy pages we should skip
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29 # (we'll bake them repeatedly later with each taxonomy term)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	30 self.taxonomy_pages = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31 logger.debug("Gathering taxonomy page paths:")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32 for tax in self.app.taxonomies:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 for src in self.app.sources:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34 path = tax.resolvePagePath(src.name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35 if path is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 self.taxonomy_pages.append(path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37 logger.debug(" - %s" % path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	38
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	39 def bake(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	40 logger.debug(" Bake Output: %s" % self.out_dir)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41 logger.debug(" Root URL: %s" % self.app.config.get('site/root'))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	43 # Get into bake mode.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44 start_time = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	45 self.app.config.set('baker/is_baking', True)
32 43091c9837bf Fix problems with asset URLs. Ludovic Chabant <ludovic@chabant.com> parents: 25 diff changeset	46 self.app.env.base_asset_url_format = '%uri%'
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	47
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	48 # Make sure the output directory exists.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 if not os.path.isdir(self.out_dir):
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	50 os.makedirs(self.out_dir, 0o755)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52 # Load/create the bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53 record = TransitionalBakeRecord()
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 42 diff changeset	54 record_cache = self.app.cache.getCache('baker')
333 91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	55 record_id = hashlib.md5(self.out_dir.encode('utf8')).hexdigest()
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	56 record_name = record_id + '.record'
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	57 if not self.force and record_cache.has(record_name):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58 t = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 record.loadPrevious(record_cache.getCachePath(record_name))
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	60 logger.debug(format_timed(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	61 t, 'loaded previous bake record',
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	62 colored=False))
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	63 record.current.success = True
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	65 # Figure out if we need to clean the cache because important things
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	66 # have changed.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	67 self._handleCacheValidity(record)
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	68
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69 # Gather all sources by realm -- we're going to bake each realm
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	70 # separately so we can handle "overlaying" (i.e. one realm overrides
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71 # another realm's pages).
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	72 sources_by_realm = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	73 for source in self.app.sources:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	74 srclist = sources_by_realm.setdefault(source.realm, [])
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75 srclist.append(source)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	77 # Bake the realms.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	78 realm_list = [REALM_USER, REALM_THEME]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	79 for realm in realm_list:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80 srclist = sources_by_realm.get(realm)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	81 if srclist is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	82 self._bakeRealm(record, realm, srclist)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	83
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84 # Bake taxonomies.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	85 self._bakeTaxonomies(record)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	86
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	87 # Delete files from the output.
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	88 self._handleDeletetions(record)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	89
333 91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	90 # Backup previous records.
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	91 for i in range(8, -1, -1):
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	92 suffix = '' if i == 0 else '.%d' % i
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	93 record_path = record_cache.getCachePath(
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	94 '%s%s.record' % (record_id, suffix))
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	95 if os.path.exists(record_path):
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	96 record_path_next = record_cache.getCachePath(
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	97 '%s.%s.record' % (record_id, i + 1))
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	98 if os.path.exists(record_path_next):
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	99 os.remove(record_path_next)
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	100 os.rename(record_path, record_path_next)
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	101
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	102 # Save the bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	103 t = time.clock()
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	104 record.current.bake_time = time.time()
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	105 record.current.out_dir = self.out_dir
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	106 record.collapseRecords()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	107 record.saveCurrent(record_cache.getCachePath(record_name))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	108 logger.debug(format_timed(t, 'saved bake record', colored=False))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	109
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	110 # All done.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	111 self.app.config.set('baker/is_baking', False)
151 fd146f54bdaa Forgot this wasn't C++. Ludovic Chabant <ludovic@chabant.com> parents: 150 diff changeset	112 logger.debug(format_timed(start_time, 'done baking'))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	113
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	114 return record.detach()
1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	115
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	116 def _handleCacheValidity(self, record):
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	117 start_time = time.clock()
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	118
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	119 reason = None
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	120 if self.force:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	121 reason = "ordered to"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	122 elif not self.app.config.get('__cache_valid'):
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	123 # The configuration file was changed, or we're running a new
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	124 # version of the app.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	125 reason = "not valid anymore"
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	126 elif (not record.previous.bake_time or
e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	127 not record.previous.hasLatestVersion()):
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	128 # We have no valid previous bake record.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	129 reason = "need bake record regeneration"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	130 else:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	131 # Check if any template has changed since the last bake. Since
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	132 # there could be some advanced conditional logic going on, we'd
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	133 # better just force a bake from scratch if that's the case.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	134 max_time = 0
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	135 for d in self.app.templates_dirs:
42 9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	136 for dpath, _, filenames in os.walk(d):
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	137 for fn in filenames:
42 9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	138 full_fn = os.path.join(dpath, fn)
9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	139 max_time = max(max_time, os.path.getmtime(full_fn))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	140 if max_time >= record.previous.bake_time:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	141 reason = "templates modified"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	142
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	143 if reason is not None:
85 3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests. Ludovic Chabant <ludovic@chabant.com> parents: 50 diff changeset	144 # We have to bake everything from scratch.
105 7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	145 for cache_name in self.app.cache.getCacheNames(
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	146 except_names=['app']):
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	147 cache_dir = self.app.cache.getCacheDir(cache_name)
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	148 if os.path.isdir(cache_dir):
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	149 logger.debug("Cleaning baker cache: %s" % cache_dir)
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	150 shutil.rmtree(cache_dir)
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 42 diff changeset	151 self.force = True
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	152 record.incremental_count = 0
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 91 diff changeset	153 record.clearPrevious()
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	154 logger.info(format_timed(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	155 start_time,
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	156 "cleaned cache (reason: %s)" % reason))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	157 else:
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	158 record.incremental_count += 1
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	159 logger.debug(format_timed(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	160 start_time, "cache is assumed valid",
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	161 colored=False))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	162
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	163 def _bakeRealm(self, record, realm, srclist):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	164 # Gather all page factories from the sources and queue them
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	165 # for the workers to pick up. Just skip taxonomy pages for now.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	166 logger.debug("Baking realm %s" % REALM_NAMES[realm])
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	167 pool, queue, abort = self._createWorkerPool(record, self.num_workers)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	168
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	169 for source in srclist:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	170 factories = source.getPageFactories()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	171 for fac in factories:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	172 if fac.path in self.taxonomy_pages:
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	173 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	174 "Skipping taxonomy page: %s:%s" %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	175 (source.name, fac.ref_spec))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	176 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	177
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	178 entry = BakeRecordPageEntry(fac.source.name, fac.rel_path,
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	179 fac.path)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	180 record.addEntry(entry)
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	181
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	182 route = self.app.getRoute(source.name, fac.metadata)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	183 if route is None:
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	184 entry.errors.append(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	185 "Can't get route for page: %s" % fac.ref_spec)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	186 logger.error(entry.errors[-1])
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	187 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	188
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	189 queue.addJob(BakeWorkerJob(fac, route, entry))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	190
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	191 success = self._waitOnWorkerPool(pool, abort)
1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	192 record.current.success &= success
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	193
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	194 def _bakeTaxonomies(self, record):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	195 logger.debug("Baking taxonomies")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	196
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	197 class _TaxonomyTermsInfo(object):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	198 def __init__(self):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	199 self.dirty_terms = set()
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	200 self.all_terms = set()
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	201
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	202 def __str__(self):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	203 return 'dirty:%s, all:%s' % (self.dirty_terms, self.all_terms)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	204
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	205 def __repr__(self):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	206 return 'dirty:%s, all:%s' % (self.dirty_terms, self.all_terms)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	207
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	208 # Let's see all the taxonomy terms for which we must bake a
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	209 # listing page... first, pre-populate our big map of used terms.
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	210 # For each source name, we have a list of taxonomies, and for each
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	211 # taxonomies, a list of terms, some being 'dirty', some used last
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	212 # time, etc.
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	213 buckets = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	214 tax_names = [t.name for t in self.app.taxonomies]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	215 source_names = [s.name for s in self.app.sources]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	216 for sn in source_names:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	217 source_taxonomies = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	218 buckets[sn] = source_taxonomies
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	219 for tn in tax_names:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	220 source_taxonomies[tn] = _TaxonomyTermsInfo()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	221
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	222 # Now see which ones are 'dirty' based on our bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	223 logger.debug("Gathering dirty taxonomy terms")
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	224 for prev_entry, cur_entry in record.transitions.values():
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	225 for tax in self.app.taxonomies:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	226 # Re-bake all taxonomy pages that include new or changed
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	227 # pages.
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	228 if cur_entry and cur_entry.was_baked_successfully:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	229 if prev_entry and prev_entry.was_baked_successfully:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	230 # Entry was re-baked this time. Mark as dirty both the
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	231 # old and new terms.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	232 changed_terms = []
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	233 prev_terms = prev_entry.config.get(tax.setting_name)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	234 cur_terms = cur_entry.config.get(tax.setting_name)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	235 if tax.is_multiple:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	236 if prev_terms is not None:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	237 changed_terms += prev_terms
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	238 if cur_terms is not None:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	239 changed_terms += cur_terms
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	240 else:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	241 if prev_terms is not None:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	242 changed_terms.append(prev_terms)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	243 if cur_terms is not None:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	244 changed_terms.append(cur_terms)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	245 else:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	246 # Entry was not baked last time. Just mark as dirty
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	247 # all the new terms.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	248 changed_terms = cur_entry.config.get(tax.setting_name)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	249
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	250 if changed_terms is not None:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	251 if not isinstance(changed_terms, list):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	252 changed_terms = [changed_terms]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	253 tt_info = buckets[cur_entry.source_name][tax.name]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	254 tt_info.dirty_terms \|= set(changed_terms)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	255
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	256 # Remember all terms used.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	257 if cur_entry and cur_entry.was_baked_successfully:
324 65e6d72f3877 bake/serve: Fix how taxonomy index pages are setup and rendered. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	258 cur_terms = cur_entry.config.get(tax.setting_name)
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	259 if cur_terms is not None:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	260 if not isinstance(cur_terms, list):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	261 cur_terms = [cur_terms]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	262 tt_info = buckets[cur_entry.source_name][tax.name]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	263 tt_info.all_terms \|= set(cur_terms)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	264 elif (prev_entry and prev_entry.was_baked_successfully and
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	265 cur_entry and not cur_entry.was_baked):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	266 prev_terms = prev_entry.config.get(tax.setting_name)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	267 if prev_terms is not None:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	268 if not isinstance(prev_terms, list):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	269 prev_terms = [prev_terms]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	270 tt_info = buckets[prev_entry.source_name][tax.name]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	271 tt_info.all_terms \|= set(prev_terms)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	272
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	273 # Re-bake the combination pages for terms that are 'dirty'.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	274 known_combinations = set()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	275 logger.debug("Gathering dirty term combinations")
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	276 for prev_entry, cur_entry in record.transitions.values():
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	277 if cur_entry and cur_entry.was_baked_successfully:
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	278 known_combinations \|= cur_entry.used_taxonomy_terms
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	279 elif prev_entry:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	280 known_combinations \|= prev_entry.used_taxonomy_terms
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	281 for sn, tn, terms in known_combinations:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	282 tt_info = buckets[sn][tn]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	283 tt_info.all_terms.add(terms)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	284 if not tt_info.dirty_terms.isdisjoint(set(terms)):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	285 tt_info.dirty_terms.add(terms)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	286
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	287 # Start baking those terms.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	288 pool, queue, abort = self._createWorkerPool(record, self.num_workers)
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	289 for source_name, source_taxonomies in buckets.items():
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	290 for tax_name, tt_info in source_taxonomies.items():
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	291 terms = tt_info.dirty_terms
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	292 if len(terms) == 0:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	293 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	294
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	295 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	296 "Baking '%s' for source '%s': %s" %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	297 (tax_name, source_name, terms))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	298 tax = self.app.getTaxonomy(tax_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	299 route = self.app.getTaxonomyRoute(tax_name, source_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	300 tax_page_ref = tax.getPageRef(source_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	301 if not tax_page_ref.exists:
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	302 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	303 "No taxonomy page found at '%s', skipping." %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	304 tax.page_ref)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	305 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	306
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	307 tax_page_source = tax_page_ref.source
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	308 tax_page_rel_path = tax_page_ref.rel_path
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	309 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	310 "Using taxonomy page: %s:%s" %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	311 (tax_page_source.name, tax_page_rel_path))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	312
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	313 for term in terms:
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	314 fac = PageFactory(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	315 tax_page_source, tax_page_rel_path,
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	316 {tax.term_name: term})
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	317 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	318 "Queuing: %s [%s, %s]" %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	319 (fac.ref_spec, tax_name, term))
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	320 entry = BakeRecordPageEntry(
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	321 fac.source.name, fac.rel_path, fac.path,
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	322 (tax_name, term, source_name))
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	323 record.addEntry(entry)
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	324 queue.addJob(BakeWorkerJob(fac, route, entry))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	325
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	326 success = self._waitOnWorkerPool(pool, abort)
1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	327 record.current.success &= success
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	328
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	329 # Now we create bake entries for all the terms that were not dirty.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	330 # This is because otherwise, on the next incremental bake, we wouldn't
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	331 # find any entry for those things, and figure that we need to delete
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	332 # their outputs.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	333 for prev_entry, cur_entry in record.transitions.values():
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	334 # Only consider taxonomy-related entries that don't have any
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	335 # current version.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	336 if (prev_entry and prev_entry.taxonomy_info and
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	337 not cur_entry):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	338 sn = prev_entry.source_name
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	339 tn, tt, tsn = prev_entry.taxonomy_info
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	340 tt_info = buckets[tsn][tn]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	341 if tt in tt_info.all_terms:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	342 logger.debug("Creating unbaked entry for taxonomy "
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	343 "term '%s:%s'." % (tn, tt))
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	344 entry = BakeRecordPageEntry(
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	345 prev_entry.source_name, prev_entry.rel_path,
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	346 prev_entry.path, prev_entry.taxonomy_info)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	347 record.addEntry(entry)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	348 else:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	349 logger.debug("Taxonomy term '%s:%s' isn't used anymore." %
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	350 (tn, tt))
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	351
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	352 def _handleDeletetions(self, record):
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	353 for path, reason in record.getDeletions():
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	354 logger.debug("Removing '%s': %s" % (path, reason))
146 0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	355 try:
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	356 os.remove(path)
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	357 logger.info('[delete] %s' % path)
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	358 except OSError:
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	359 # Not a big deal if that file had already been removed
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	360 # by the user.
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	361 pass
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	362
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	363 def _createWorkerPool(self, record, pool_size=4):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	364 pool = []
85 3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests. Ludovic Chabant <ludovic@chabant.com> parents: 50 diff changeset	365 queue = BakeScheduler(record)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	366 abort = threading.Event()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	367 for i in range(pool_size):
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	368 ctx = BakeWorkerContext(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	369 self.app, self.out_dir, self.force,
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	370 record, queue, abort)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	371 worker = BakeWorker(i, ctx)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	372 pool.append(worker)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	373 return pool, queue, abort
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	374
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	375 def _waitOnWorkerPool(self, pool, abort):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	376 for w in pool:
85 3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests. Ludovic Chabant <ludovic@chabant.com> parents: 50 diff changeset	377 w.start()
159 232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	378
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	379 success = True
159 232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	380 try:
232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	381 for w in pool:
232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	382 w.join()
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	383 success &= w.success
159 232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	384 except KeyboardInterrupt:
232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	385 logger.warning("Bake aborted by user... "
232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	386 "waiting for workers to stop.")
232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	387 abort.set()
232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	388 for w in pool:
232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	389 w.join()
232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	390 raise
232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	391
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	392 if abort.is_set():
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	393 excs = [w.abort_exception for w in pool
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	394 if w.abort_exception is not None]
85 3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests. Ludovic Chabant <ludovic@chabant.com> parents: 50 diff changeset	395 logger.error("Baking was aborted due to %s error(s):" % len(excs))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	396 if self.app.debug:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	397 for e in excs:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	398 logger.exception(e)
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	399 else:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	400 for e in excs:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	401 log_friendly_exception(logger, e)
105 7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	402 raise BakingError("Baking was aborted due to errors.")
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	403
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	404 return success
1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	405
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	406
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	407 class BakeWorkerContext(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	408 def __init__(self, app, out_dir, force, record, work_queue,
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	409 abort_event):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	410 self.app = app
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	411 self.out_dir = out_dir
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	412 self.force = force
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	413 self.record = record
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	414 self.work_queue = work_queue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	415 self.abort_event = abort_event
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	416
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	417
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	418 class BakeWorkerJob(object):
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	419 def __init__(self, factory, route, record_entry):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	420 self.factory = factory
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	421 self.route = route
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	422 self.record_entry = record_entry
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	423
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	424 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	425 def source(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	426 return self.factory.source
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	427
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	428
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	429 class BakeWorker(threading.Thread):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	430 def __init__(self, wid, ctx):
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	431 super(BakeWorker, self).__init__(name=('worker%d' % wid))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	432 self.wid = wid
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	433 self.ctx = ctx
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	434 self.abort_exception = None
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	435 self.success = True
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	436 self._page_baker = PageBaker(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	437 ctx.app, ctx.out_dir, ctx.force,
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	438 ctx.record)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	439
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	440 def run(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	441 while(not self.ctx.abort_event.is_set()):
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 91 diff changeset	442 try:
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 91 diff changeset	443 job = self.ctx.work_queue.getNextJob(wait_timeout=1)
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 91 diff changeset	444 if job is None:
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	445 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	446 "[%d] No more work... shutting down." %
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 91 diff changeset	447 self.wid)
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 91 diff changeset	448 break
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	449 success = self._unsafeRun(job)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	450 logger.debug("[%d] Done with page." % self.wid)
85 3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests. Ludovic Chabant <ludovic@chabant.com> parents: 50 diff changeset	451 self.ctx.work_queue.onJobFinished(job)
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	452 self.success &= success
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	453 except Exception as ex:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	454 self.ctx.abort_event.set()
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	455 self.abort_exception = ex
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	456 self.success = False
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	457 logger.debug("[%d] Critical error, aborting." % self.wid)
50 2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	458 if self.ctx.app.debug:
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	459 logger.exception(ex)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	460 break
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	461
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	462 def _unsafeRun(self, job):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	463 start_time = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	464
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	465 entry = job.record_entry
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	466 try:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	467 self._page_baker.bake(job.factory, job.route, entry)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	468 except BakingError as ex:
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	469 logger.debug("Got baking error. Adding it to the record.")
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	470 while ex:
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	471 entry.errors.append(str(ex))
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	472 ex = ex.__cause__
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	473
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	474 if entry.errors:
1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	475 for e in entry.errors:
1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	476 logger.error(e)
1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	477 return False
1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	478
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	479 if entry.was_baked_successfully:
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	480 uri = entry.out_uris[0]
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	481 friendly_uri = uri if uri != '' else '[main page]'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	482 friendly_count = ''
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	483 if entry.num_subs > 1:
e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	484 friendly_count = ' (%d pages)' % entry.num_subs
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	485 logger.info(format_timed(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	486 start_time, '[%d] %s%s' %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	487 (self.wid, friendly_uri, friendly_count)))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	488
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	489 return True
1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	490

Mercurial > piecrust2

annotate piecrust/baking/baker.py @ 334:b034f6f15e22