piecrust2: piecrust/baking/baker.py annotate

annotate piecrust/baking/baker.py @ 413:eacf0a3afd0c

internal: Register performance timers for plugin components.

author	Ludovic Chabant <ludovic@chabant.com>
date	Sat, 20 Jun 2015 19:16:38 -0700
parents	e7b865f8f335
children	0e9a94b7fdfa

rev	line source
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	1 import copy
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import time
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import os.path
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	4 import queue
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import hashlib
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 import logging
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	7 import multiprocessing
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	8 from piecrust.baking.records import (
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	9 BakeRecordEntry, TransitionalBakeRecord, TaxonomyInfo, FirstRenderInfo)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	10 from piecrust.baking.worker import (
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	11 BakeWorkerJob, LoadJobPayload, RenderFirstSubJobPayload,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	12 BakeJobPayload,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	13 JOB_LOAD, JOB_RENDER_FIRST, JOB_BAKE)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	14 from piecrust.chefutil import (
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	15 format_timed_scope, format_timed)
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	16 from piecrust.sources.base import (
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17 REALM_NAMES, REALM_USER, REALM_THEME)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	23 class Baker(object):
205 e725af1d48fb bake: Changes in how assets directories are configured. Ludovic Chabant <ludovic@chabant.com> parents: 199 diff changeset	24 def __init__(self, app, out_dir, force=False):
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	25 assert app and out_dir
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26 self.app = app
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	27 self.out_dir = out_dir
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 self.force = force
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	29 self.num_workers = app.config.get('baker/workers',
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	30 multiprocessing.cpu_count())
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32 # Remember what taxonomy pages we should skip
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 # (we'll bake them repeatedly later with each taxonomy term)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34 self.taxonomy_pages = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35 logger.debug("Gathering taxonomy page paths:")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 for tax in self.app.taxonomies:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37 for src in self.app.sources:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	38 tax_page_ref = tax.getPageRef(src)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	39 for path in tax_page_ref.possible_paths:
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	40 self.taxonomy_pages.append(path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41 logger.debug(" - %s" % path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	43 def bake(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44 logger.debug(" Bake Output: %s" % self.out_dir)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	45 logger.debug(" Root URL: %s" % self.app.config.get('site/root'))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	46
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	47 # Get into bake mode.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	48 start_time = time.perf_counter()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 self.app.config.set('baker/is_baking', True)
32 43091c9837bf Fix problems with asset URLs. Ludovic Chabant <ludovic@chabant.com> parents: 25 diff changeset	50 self.app.env.base_asset_url_format = '%uri%'
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52 # Make sure the output directory exists.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53 if not os.path.isdir(self.out_dir):
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	54 os.makedirs(self.out_dir, 0o755)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	55
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	56 # Load/create the bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	57 record = TransitionalBakeRecord()
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 42 diff changeset	58 record_cache = self.app.cache.getCache('baker')
333 91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	59 record_id = hashlib.md5(self.out_dir.encode('utf8')).hexdigest()
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	60 record_name = record_id + '.record'
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	61 previous_record_path = None
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62 if not self.force and record_cache.has(record_name):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	63 with format_timed_scope(logger, "loaded previous bake record",
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	64 level=logging.DEBUG, colored=False):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	65 previous_record_path = record_cache.getCachePath(record_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	66 record.loadPrevious(previous_record_path)
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	67 record.current.success = True
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	69 # Figure out if we need to clean the cache because important things
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	70 # have changed.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	71 self._handleCacheValidity(record)
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	72
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	73 # Pre-create all caches.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	74 for cache_name in ['app', 'baker', 'pages', 'renders']:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	75 self.app.cache.getCache(cache_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	76
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	77 # Gather all sources by realm -- we're going to bake each realm
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	78 # separately so we can handle "overriding" (i.e. one realm overrides
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	79 # another realm's pages, like the user realm overriding the theme
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	80 # realm).
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	81 sources_by_realm = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	82 for source in self.app.sources:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	83 srclist = sources_by_realm.setdefault(source.realm, [])
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84 srclist.append(source)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	85
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	86 # Create the worker processes.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	87 pool = self._createWorkerPool()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	88
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	89 # Bake the realms.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	90 realm_list = [REALM_USER, REALM_THEME]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	91 for realm in realm_list:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	92 srclist = sources_by_realm.get(realm)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	93 if srclist is not None:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	94 self._bakeRealm(record, pool, realm, srclist)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	95
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	96 # Bake taxonomies.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	97 self._bakeTaxonomies(record, pool)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	98
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	99 # All done with the workers.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	100 self._terminateWorkerPool(pool)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	101
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	102 # Get the timing information from the workers.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	103 record.current.timers = {}
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	104 for _ in range(len(pool.workers)):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	105 try:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	106 timers = pool.results.get(True, 0.1)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	107 except queue.Empty:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	108 logger.error("Didn't get timing information from all workers.")
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	109 break
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	110
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	111 for name, val in timers.items():
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	112 main_val = record.current.timers.setdefault(name, 0)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	113 record.current.timers[name] = main_val + val
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	114
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	115 # Delete files from the output.
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	116 self._handleDeletetions(record)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	117
333 91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	118 # Backup previous records.
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	119 for i in range(8, -1, -1):
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	120 suffix = '' if i == 0 else '.%d' % i
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	121 record_path = record_cache.getCachePath(
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	122 '%s%s.record' % (record_id, suffix))
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	123 if os.path.exists(record_path):
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	124 record_path_next = record_cache.getCachePath(
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	125 '%s.%s.record' % (record_id, i + 1))
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	126 if os.path.exists(record_path_next):
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	127 os.remove(record_path_next)
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	128 os.rename(record_path, record_path_next)
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	129
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	130 # Save the bake record.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	131 with format_timed_scope(logger, "saved bake record.",
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	132 level=logging.DEBUG, colored=False):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	133 record.current.bake_time = time.time()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	134 record.current.out_dir = self.out_dir
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	135 record.saveCurrent(record_cache.getCachePath(record_name))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	136
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	137 # All done.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	138 self.app.config.set('baker/is_baking', False)
151 fd146f54bdaa Forgot this wasn't C++. Ludovic Chabant <ludovic@chabant.com> parents: 150 diff changeset	139 logger.debug(format_timed(start_time, 'done baking'))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	140
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	141 return record.detach()
1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	142
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	143 def _handleCacheValidity(self, record):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	144 start_time = time.perf_counter()
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	145
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	146 reason = None
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	147 if self.force:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	148 reason = "ordered to"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	149 elif not self.app.config.get('__cache_valid'):
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	150 # The configuration file was changed, or we're running a new
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	151 # version of the app.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	152 reason = "not valid anymore"
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	153 elif (not record.previous.bake_time or
e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	154 not record.previous.hasLatestVersion()):
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	155 # We have no valid previous bake record.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	156 reason = "need bake record regeneration"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	157 else:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	158 # Check if any template has changed since the last bake. Since
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	159 # there could be some advanced conditional logic going on, we'd
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	160 # better just force a bake from scratch if that's the case.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	161 max_time = 0
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	162 for d in self.app.templates_dirs:
42 9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	163 for dpath, _, filenames in os.walk(d):
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	164 for fn in filenames:
42 9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	165 full_fn = os.path.join(dpath, fn)
9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	166 max_time = max(max_time, os.path.getmtime(full_fn))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	167 if max_time >= record.previous.bake_time:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	168 reason = "templates modified"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	169
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	170 if reason is not None:
85 3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests. Ludovic Chabant <ludovic@chabant.com> parents: 50 diff changeset	171 # We have to bake everything from scratch.
371 c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	172 self.app.cache.clearCaches(except_names=['app'])
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 42 diff changeset	173 self.force = True
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	174 record.incremental_count = 0
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 91 diff changeset	175 record.clearPrevious()
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	176 logger.info(format_timed(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	177 start_time,
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	178 "cleaned cache (reason: %s)" % reason))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	179 else:
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	180 record.incremental_count += 1
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	181 logger.debug(format_timed(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	182 start_time, "cache is assumed valid",
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	183 colored=False))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	184
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	185 def _bakeRealm(self, record, pool, realm, srclist):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	186 start_time = time.perf_counter()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	187 try:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	188 all_factories = []
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	189 for source in srclist:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	190 factories = source.getPageFactories()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	191 all_factories += [f for f in factories
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	192 if f.path not in self.taxonomy_pages]
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	193
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	194 self._loadRealmPages(record, pool, all_factories)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	195 self._renderRealmPages(record, pool, all_factories)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	196 self._bakeRealmPages(record, pool, all_factories)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	197 finally:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	198 page_count = len(all_factories)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	199 logger.info(format_timed(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	200 start_time,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	201 "baked %d %s pages" %
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	202 (page_count, REALM_NAMES[realm].lower())))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	203
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	204 def _loadRealmPages(self, record, pool, factories):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	205 with format_timed_scope(logger,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	206 "loaded %d pages" % len(factories),
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	207 level=logging.DEBUG, colored=False):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	208 for fac in factories:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	209 job = BakeWorkerJob(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	210 JOB_LOAD,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	211 LoadJobPayload(fac))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	212 pool.queue.put_nowait(job)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	213
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	214 def _handler(res):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	215 # Create the record entry for this page.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	216 record_entry = BakeRecordEntry(res.source_name, res.path)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	217 record_entry.config = res.config
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	218 if res.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	219 record_entry.errors += res.errors
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	220 record.current.success = False
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	221 record.addEntry(record_entry)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	222
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	223 self._waitOnWorkerPool(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	224 pool,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	225 expected_result_count=len(factories),
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	226 result_handler=_handler)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	227
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	228 def _renderRealmPages(self, record, pool, factories):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	229 with format_timed_scope(logger,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	230 "prepared %d pages" % len(factories),
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	231 level=logging.DEBUG, colored=False):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	232 expected_result_count = 0
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	233 for fac in factories:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	234 record_entry = record.getCurrentEntry(fac.path)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	235 if record_entry.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	236 logger.debug("Ignoring %s because it had previous "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	237 "errors." % fac.ref_spec)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	238 continue
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	239
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	240 # Make sure the source and the route exist for this page,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	241 # otherwise we add errors to the record entry and we'll skip
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	242 # this page for the rest of the bake.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	243 source = self.app.getSource(fac.source.name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	244 if source is None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	245 record_entry.errors.append(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	246 "Can't get source for page: %s" % fac.ref_spec)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	247 logger.error(record_entry.errors[-1])
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	248 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	249
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	250 route = self.app.getRoute(fac.source.name, fac.metadata,
363 dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 338 diff changeset	251 skip_taxonomies=True)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	252 if route is None:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	253 record_entry.errors.append(
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	254 "Can't get route for page: %s" % fac.ref_spec)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	255 logger.error(record_entry.errors[-1])
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	256 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	257
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	258 # All good, queue the job.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	259 job = BakeWorkerJob(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	260 JOB_RENDER_FIRST,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	261 RenderFirstSubJobPayload(fac))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	262 pool.queue.put_nowait(job)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	263 expected_result_count += 1
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	264
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	265 def _handler(res):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	266 entry = record.getCurrentEntry(res.path)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	267
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	268 entry.first_render_info = FirstRenderInfo()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	269 entry.first_render_info.used_assets = res.used_assets
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	270 entry.first_render_info.used_pagination = \
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	271 res.used_pagination
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	272 entry.first_render_info.pagination_has_more = \
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	273 res.pagination_has_more
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	274
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	275 if res.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	276 entry.errors += res.errors
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	277 record.current.success = False
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	278
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	279 self._waitOnWorkerPool(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	280 pool,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	281 expected_result_count=expected_result_count,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	282 result_handler=_handler)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	283
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	284 def _bakeRealmPages(self, record, pool, factories):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	285 with format_timed_scope(logger,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	286 "baked %d pages" % len(factories),
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	287 level=logging.DEBUG, colored=False):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	288 expected_result_count = 0
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	289 for fac in factories:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	290 if self._queueBakeJob(record, pool, fac):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	291 expected_result_count += 1
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	292
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	293 def _handler(res):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	294 entry = record.getCurrentEntry(res.path, res.taxonomy_info)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	295 entry.bake_info = res.bake_info
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	296 if res.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	297 entry.errors += res.errors
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	298 if entry.has_any_error:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	299 record.current.success = False
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	300
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	301 self._waitOnWorkerPool(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	302 pool,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	303 expected_result_count=expected_result_count,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	304 result_handler=_handler)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	305
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	306 def _bakeTaxonomies(self, record, pool):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	307 with format_timed_scope(logger, 'built taxonomy buckets',
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	308 level=logging.DEBUG, colored=False):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	309 buckets = self._buildTaxonomyBuckets(record)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	310
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	311 start_time = time.perf_counter()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	312 page_count = self._bakeTaxonomyBuckets(record, pool, buckets)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	313 logger.info(format_timed(start_time,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	314 "baked %d taxonomy pages." % page_count))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	315
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	316 def _buildTaxonomyBuckets(self, record):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	317 # Let's see all the taxonomy terms for which we must bake a
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	318 # listing page... first, pre-populate our big map of used terms.
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	319 # For each source name, we have a list of taxonomies, and for each
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	320 # taxonomies, a list of terms, some being 'dirty', some used last
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	321 # time, etc.
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	322 buckets = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	323 tax_names = [t.name for t in self.app.taxonomies]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	324 source_names = [s.name for s in self.app.sources]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	325 for sn in source_names:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	326 source_taxonomies = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	327 buckets[sn] = source_taxonomies
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	328 for tn in tax_names:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	329 source_taxonomies[tn] = _TaxonomyTermsInfo()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	330
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	331 # Now see which ones are 'dirty' based on our bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	332 logger.debug("Gathering dirty taxonomy terms")
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	333 for prev_entry, cur_entry in record.transitions.values():
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	334 # Re-bake all taxonomy pages that include new or changed
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	335 # pages.
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	336 if cur_entry and cur_entry.was_any_sub_baked:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	337 entries = [cur_entry]
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	338 if prev_entry:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	339 entries.append(prev_entry)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	340
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	341 for tax in self.app.taxonomies:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	342 changed_terms = set()
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	343 for e in entries:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	344 terms = e.config.get(tax.setting_name)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	345 if terms:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	346 if not tax.is_multiple:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	347 terms = [terms]
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	348 changed_terms \|= set(terms)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	349
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	350 if len(changed_terms) > 0:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	351 tt_info = buckets[cur_entry.source_name][tax.name]
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	352 tt_info.dirty_terms \|= changed_terms
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	353
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	354 # Remember all terms used.
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	355 for tax in self.app.taxonomies:
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	356 if cur_entry and not cur_entry.was_overriden:
324 65e6d72f3877 bake/serve: Fix how taxonomy index pages are setup and rendered. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	357 cur_terms = cur_entry.config.get(tax.setting_name)
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	358 if cur_terms:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	359 if not tax.is_multiple:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	360 cur_terms = [cur_terms]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	361 tt_info = buckets[cur_entry.source_name][tax.name]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	362 tt_info.all_terms \|= set(cur_terms)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	363
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	364 # Re-bake the combination pages for terms that are 'dirty'.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	365 known_combinations = set()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	366 logger.debug("Gathering dirty term combinations")
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	367 for prev_entry, cur_entry in record.transitions.values():
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	368 if not cur_entry:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	369 continue
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	370 used_taxonomy_terms = cur_entry.getAllUsedTaxonomyTerms()
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	371 for sn, tn, terms in used_taxonomy_terms:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	372 if isinstance(terms, tuple):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	373 known_combinations.add((sn, tn, terms))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	374 for sn, tn, terms in known_combinations:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	375 tt_info = buckets[sn][tn]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	376 tt_info.all_terms.add(terms)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	377 if not tt_info.dirty_terms.isdisjoint(set(terms)):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	378 tt_info.dirty_terms.add(terms)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	379
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	380 return buckets
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	381
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	382 def _bakeTaxonomyBuckets(self, record, pool, buckets):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	383 # Start baking those terms.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	384 expected_result_count = 0
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	385 for source_name, source_taxonomies in buckets.items():
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	386 for tax_name, tt_info in source_taxonomies.items():
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	387 terms = tt_info.dirty_terms
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	388 if len(terms) == 0:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	389 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	390
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	391 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	392 "Baking '%s' for source '%s': %s" %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	393 (tax_name, source_name, terms))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	394 tax = self.app.getTaxonomy(tax_name)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	395 source = self.app.getSource(source_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	396 tax_page_ref = tax.getPageRef(source)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	397 if not tax_page_ref.exists:
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	398 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	399 "No taxonomy page found at '%s', skipping." %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	400 tax.page_ref)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	401 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	402
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	403 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	404 "Using taxonomy page: %s:%s" %
369 4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	405 (tax_page_ref.source_name, tax_page_ref.rel_path))
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	406 fac = tax_page_ref.getFactory()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	407
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	408 for term in terms:
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	409 logger.debug(
369 4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	410 "Queuing: %s [%s=%s]" %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	411 (fac.ref_spec, tax_name, term))
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	412 tax_info = TaxonomyInfo(tax_name, source_name, term)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	413
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	414 cur_entry = BakeRecordEntry(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	415 fac.source.name, fac.path, tax_info)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	416 record.addEntry(cur_entry)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	417
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	418 if self._queueBakeJob(record, pool, fac, tax_info):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	419 expected_result_count += 1
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	420
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	421 def _handler(res):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	422 entry = record.getCurrentEntry(res.path, res.taxonomy_info)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	423 entry.bake_info = res.bake_info
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	424 if res.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	425 entry.errors += res.errors
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	426 if entry.has_any_error:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	427 record.current.success = False
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	428
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	429 self._waitOnWorkerPool(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	430 pool,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	431 expected_result_count=expected_result_count,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	432 result_handler=_handler)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	433
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	434 # Now we create bake entries for all the terms that were not dirty.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	435 # This is because otherwise, on the next incremental bake, we wouldn't
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	436 # find any entry for those things, and figure that we need to delete
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	437 # their outputs.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	438 for prev_entry, cur_entry in record.transitions.values():
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	439 # Only consider taxonomy-related entries that don't have any
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	440 # current version.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	441 if (prev_entry and prev_entry.taxonomy_info and
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	442 not cur_entry):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	443 ti = prev_entry.taxonomy_info
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	444 tt_info = buckets[ti.source_name][ti.taxonomy_name]
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	445 if ti.term in tt_info.all_terms:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	446 logger.debug("Creating unbaked entry for taxonomy "
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	447 "term '%s:%s'." % (ti.taxonomy_name, ti.term))
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	448 record.collapseEntry(prev_entry)
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	449 else:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	450 logger.debug("Taxonomy term '%s:%s' isn't used anymore." %
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	451 (ti.taxonomy_name, ti.term))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	452
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	453 return expected_result_count
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	454
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	455 def _queueBakeJob(self, record, pool, fac, tax_info=None):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	456 # Get the previous (if any) and current entry for this page.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	457 pair = record.getPreviousAndCurrentEntries(fac.path, tax_info)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	458 assert pair is not None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	459 prev_entry, cur_entry = pair
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	460 assert cur_entry is not None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	461
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	462 # Ignore if there were errors in the previous passes.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	463 if cur_entry.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	464 logger.debug("Ignoring %s because it had previous "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	465 "errors." % fac.ref_spec)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	466 return False
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	467
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	468 # Build the route metadata and find the appropriate route.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	469 route_metadata = copy.deepcopy(fac.metadata)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	470 if tax_info is not None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	471 tax = self.app.getTaxonomy(tax_info.taxonomy_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	472 route = self.app.getTaxonomyRoute(tax_info.taxonomy_name,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	473 tax_info.source_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	474
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	475 slugified_term = route.slugifyTaxonomyTerm(tax_info.term)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	476 route_metadata[tax.term_name] = slugified_term
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	477 else:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	478 route = self.app.getRoute(fac.source.name, route_metadata,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	479 skip_taxonomies=True)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	480 assert route is not None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	481
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	482 # Figure out if this page is overriden by another previously
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	483 # baked page. This happens for example when the user has
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	484 # made a page that has the same page/URL as a theme page.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	485 page = fac.buildPage()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	486 uri = route.getUri(route_metadata, provider=page)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	487 override_entry = record.getOverrideEntry(page.path, uri)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	488 if override_entry is not None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	489 override_source = self.app.getSource(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	490 override_entry.source_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	491 if override_source.realm == fac.source.realm:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	492 cur_entry.errors.append(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	493 "Page '%s' maps to URL '%s' but is overriden "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	494 "by page '%s'." %
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	495 (fac.ref_spec, uri, override_entry.path))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	496 logger.error(cur_entry.errors[-1])
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	497 cur_entry.flags \|= BakeRecordEntry.FLAG_OVERRIDEN
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	498 return False
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	499
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	500 job = BakeWorkerJob(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	501 JOB_BAKE,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	502 BakeJobPayload(fac, route_metadata, prev_entry,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	503 cur_entry.first_render_info,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	504 record.dirty_source_names,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	505 tax_info))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	506 pool.queue.put_nowait(job)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	507 return True
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	508
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	509 def _handleDeletetions(self, record):
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	510 for path, reason in record.getDeletions():
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	511 logger.debug("Removing '%s': %s" % (path, reason))
146 0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	512 try:
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	513 os.remove(path)
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	514 logger.info('[delete] %s' % path)
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	515 except OSError:
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	516 # Not a big deal if that file had already been removed
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	517 # by the user.
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	518 pass
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	519
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	520 def _createWorkerPool(self):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	521 from piecrust.baking.worker import BakeWorkerContext, worker_func
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	522
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	523 pool = _WorkerPool()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	524 for i in range(self.num_workers):
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	525 ctx = BakeWorkerContext(
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	526 self.app.root_dir, self.out_dir,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	527 pool.queue, pool.results, pool.abort_event,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	528 force=self.force, debug=self.app.debug)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	529 w = multiprocessing.Process(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	530 target=worker_func, args=(i, ctx))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	531 w.start()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	532 pool.workers.append(w)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	533 return pool
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	534
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	535 def _terminateWorkerPool(self, pool):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	536 pool.abort_event.set()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	537 for w in pool.workers:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	538 w.join()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	539
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	540 def _waitOnWorkerPool(self, pool,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	541 expected_result_count=-1, result_handler=None):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	542 assert result_handler is None or expected_result_count >= 0
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	543 abort_with_exception = None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	544 try:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	545 if result_handler is None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	546 pool.queue.join()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	547 else:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	548 got_count = 0
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	549 while got_count < expected_result_count:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	550 try:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	551 res = pool.results.get(True, 10)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	552 except queue.Empty:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	553 logger.error(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	554 "Got %d results, expected %d, and timed-out "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	555 "for 10 seconds. A worker might be stuck?" %
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	556 (got_count, expected_result_count))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	557 abort_with_exception = Exception("Worker time-out.")
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	558 break
159 232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	559
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	560 got_count += 1
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	561 result_handler(res)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	562 except KeyboardInterrupt as kiex:
159 232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	563 logger.warning("Bake aborted by user... "
232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	564 "waiting for workers to stop.")
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	565 abort_with_exception = kiex
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	566
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	567 if abort_with_exception:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	568 pool.abort_event.set()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	569 for w in pool.workers:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	570 w.join(2)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	571 raise abort_with_exception
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	572
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	573
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	574 class _WorkerPool(object):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	575 def __init__(self):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	576 self.queue = multiprocessing.JoinableQueue()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	577 self.results = multiprocessing.Queue()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	578 self.abort_event = multiprocessing.Event()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	579 self.workers = []
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	580
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	581
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	582 class _TaxonomyTermsInfo(object):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	583 def __init__(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	584 self.dirty_terms = set()
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	585 self.all_terms = set()
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	586
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	587 def __str__(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	588 return 'dirty:%s, all:%s' % (self.dirty_terms, self.all_terms)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	589
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	590 def __repr__(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	591 return 'dirty:%s, all:%s' % (self.dirty_terms, self.all_terms)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	592

Mercurial > piecrust2

annotate piecrust/baking/baker.py @ 413:eacf0a3afd0c