piecrust2: piecrust/baking/baker.py annotate

annotate piecrust/baking/baker.py @ 440:32c7c2d219d2

performance: Refactor how data is managed to reduce copying. * Make use of `collections.abc.Mapping` to better identify things that are supposed to look like dictionaries. * Instead of handling "overlay" of data in a dict tree in each different data object, make all objects `Mapping`s and handle merging at a higher level with the new `MergedMapping` object. * Since this new object is read-only, remove the need for deep-copying of app and page configurations. * Split data classes into separate modules.

author	Ludovic Chabant <ludovic@chabant.com>
date	Sun, 28 Jun 2015 08:22:39 -0700
parents	21e26ed867b6
children	aefe70229fdd

rev	line source
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	1 import copy
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import time
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import os.path
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	4 import queue
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import hashlib
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 import logging
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	7 import multiprocessing
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	8 from piecrust.baking.records import (
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	9 BakeRecordEntry, TransitionalBakeRecord, TaxonomyInfo)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	10 from piecrust.baking.worker import (
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	11 BakeWorkerJob, LoadJobPayload, RenderFirstSubJobPayload,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	12 BakeJobPayload,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	13 JOB_LOAD, JOB_RENDER_FIRST, JOB_BAKE)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	14 from piecrust.chefutil import (
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	15 format_timed_scope, format_timed)
430 21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	16 from piecrust.routing import create_route_metadata
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	17 from piecrust.sources.base import (
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18 REALM_NAMES, REALM_USER, REALM_THEME)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	23
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24 class Baker(object):
205 e725af1d48fb bake: Changes in how assets directories are configured. Ludovic Chabant <ludovic@chabant.com> parents: 199 diff changeset	25 def __init__(self, app, out_dir, force=False):
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	26 assert app and out_dir
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27 self.app = app
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	28 self.out_dir = out_dir
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29 self.force = force
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	30 self.num_workers = app.config.get('baker/workers',
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	31 multiprocessing.cpu_count())
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 # Remember what taxonomy pages we should skip
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34 # (we'll bake them repeatedly later with each taxonomy term)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35 self.taxonomy_pages = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 logger.debug("Gathering taxonomy page paths:")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37 for tax in self.app.taxonomies:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	38 for src in self.app.sources:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	39 tax_page_ref = tax.getPageRef(src)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	40 for path in tax_page_ref.possible_paths:
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41 self.taxonomy_pages.append(path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42 logger.debug(" - %s" % path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	43
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	44 # Register some timers.
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	45 self.app.env.registerTimer('LoadJob', raise_if_registered=False)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	46 self.app.env.registerTimer('RenderFirstSubJob',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	47 raise_if_registered=False)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	48 self.app.env.registerTimer('BakeJob', raise_if_registered=False)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	49
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50 def bake(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51 logger.debug(" Bake Output: %s" % self.out_dir)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52 logger.debug(" Root URL: %s" % self.app.config.get('site/root'))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54 # Get into bake mode.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	55 start_time = time.perf_counter()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	56 self.app.config.set('baker/is_baking', True)
32 43091c9837bf Fix problems with asset URLs. Ludovic Chabant <ludovic@chabant.com> parents: 25 diff changeset	57 self.app.env.base_asset_url_format = '%uri%'
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 # Make sure the output directory exists.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	60 if not os.path.isdir(self.out_dir):
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	61 os.makedirs(self.out_dir, 0o755)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63 # Load/create the bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64 record = TransitionalBakeRecord()
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 42 diff changeset	65 record_cache = self.app.cache.getCache('baker')
333 91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	66 record_id = hashlib.md5(self.out_dir.encode('utf8')).hexdigest()
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	67 record_name = record_id + '.record'
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	68 previous_record_path = None
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69 if not self.force and record_cache.has(record_name):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	70 with format_timed_scope(logger, "loaded previous bake record",
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	71 level=logging.DEBUG, colored=False):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	72 previous_record_path = record_cache.getCachePath(record_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	73 record.loadPrevious(previous_record_path)
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	74 record.current.success = True
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	76 # Figure out if we need to clean the cache because important things
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	77 # have changed.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	78 self._handleCacheValidity(record)
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	79
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	80 # Pre-create all caches.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	81 for cache_name in ['app', 'baker', 'pages', 'renders']:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	82 self.app.cache.getCache(cache_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	83
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84 # Gather all sources by realm -- we're going to bake each realm
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	85 # separately so we can handle "overriding" (i.e. one realm overrides
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	86 # another realm's pages, like the user realm overriding the theme
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	87 # realm).
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	88 sources_by_realm = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	89 for source in self.app.sources:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	90 srclist = sources_by_realm.setdefault(source.realm, [])
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	91 srclist.append(source)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	92
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	93 # Create the worker processes.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	94 pool = self._createWorkerPool()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	95
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	96 # Bake the realms.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	97 realm_list = [REALM_USER, REALM_THEME]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	98 for realm in realm_list:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	99 srclist = sources_by_realm.get(realm)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	100 if srclist is not None:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	101 self._bakeRealm(record, pool, realm, srclist)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	102
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	103 # Bake taxonomies.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	104 self._bakeTaxonomies(record, pool)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	105
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	106 # All done with the workers.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	107 self._terminateWorkerPool(pool)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	108
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	109 # Get the timing information from the workers.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	110 record.current.timers = {}
421 4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	111 for i in range(len(pool.workers)):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	112 try:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	113 timers = pool.results.get(True, 0.1)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	114 except queue.Empty:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	115 logger.error("Didn't get timing information from all workers.")
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	116 break
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	117
421 4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	118 worker_name = 'BakeWorker_%d' % i
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	119 record.current.timers[worker_name] = {}
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	120 for name, val in timers['data'].items():
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	121 main_val = record.current.timers.setdefault(name, 0)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	122 record.current.timers[name] = main_val + val
421 4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	123 record.current.timers[worker_name][name] = val
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	124
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	125 # Delete files from the output.
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	126 self._handleDeletetions(record)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	127
333 91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	128 # Backup previous records.
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	129 for i in range(8, -1, -1):
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	130 suffix = '' if i == 0 else '.%d' % i
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	131 record_path = record_cache.getCachePath(
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	132 '%s%s.record' % (record_id, suffix))
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	133 if os.path.exists(record_path):
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	134 record_path_next = record_cache.getCachePath(
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	135 '%s.%s.record' % (record_id, i + 1))
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	136 if os.path.exists(record_path_next):
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	137 os.remove(record_path_next)
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	138 os.rename(record_path, record_path_next)
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	139
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	140 # Save the bake record.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	141 with format_timed_scope(logger, "saved bake record.",
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	142 level=logging.DEBUG, colored=False):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	143 record.current.bake_time = time.time()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	144 record.current.out_dir = self.out_dir
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	145 record.saveCurrent(record_cache.getCachePath(record_name))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	146
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	147 # All done.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	148 self.app.config.set('baker/is_baking', False)
151 fd146f54bdaa Forgot this wasn't C++. Ludovic Chabant <ludovic@chabant.com> parents: 150 diff changeset	149 logger.debug(format_timed(start_time, 'done baking'))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	151 return record.detach()
1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	152
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	153 def _handleCacheValidity(self, record):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	154 start_time = time.perf_counter()
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	155
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	156 reason = None
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	157 if self.force:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	158 reason = "ordered to"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	159 elif not self.app.config.get('__cache_valid'):
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	160 # The configuration file was changed, or we're running a new
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	161 # version of the app.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	162 reason = "not valid anymore"
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	163 elif (not record.previous.bake_time or
e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	164 not record.previous.hasLatestVersion()):
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	165 # We have no valid previous bake record.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	166 reason = "need bake record regeneration"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	167 else:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	168 # Check if any template has changed since the last bake. Since
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	169 # there could be some advanced conditional logic going on, we'd
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	170 # better just force a bake from scratch if that's the case.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	171 max_time = 0
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	172 for d in self.app.templates_dirs:
42 9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	173 for dpath, _, filenames in os.walk(d):
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	174 for fn in filenames:
42 9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	175 full_fn = os.path.join(dpath, fn)
9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	176 max_time = max(max_time, os.path.getmtime(full_fn))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	177 if max_time >= record.previous.bake_time:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	178 reason = "templates modified"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	179
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	180 if reason is not None:
85 3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests. Ludovic Chabant <ludovic@chabant.com> parents: 50 diff changeset	181 # We have to bake everything from scratch.
371 c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	182 self.app.cache.clearCaches(except_names=['app'])
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 42 diff changeset	183 self.force = True
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	184 record.incremental_count = 0
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 91 diff changeset	185 record.clearPrevious()
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	186 logger.info(format_timed(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	187 start_time,
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	188 "cleaned cache (reason: %s)" % reason))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	189 else:
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	190 record.incremental_count += 1
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	191 logger.debug(format_timed(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	192 start_time, "cache is assumed valid",
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	193 colored=False))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	194
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	195 def _bakeRealm(self, record, pool, realm, srclist):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	196 start_time = time.perf_counter()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	197 try:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	198 record.current.baked_count[realm] = 0
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	199
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	200 all_factories = []
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	201 for source in srclist:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	202 factories = source.getPageFactories()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	203 all_factories += [f for f in factories
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	204 if f.path not in self.taxonomy_pages]
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	205
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	206 self._loadRealmPages(record, pool, all_factories)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	207 self._renderRealmPages(record, pool, all_factories)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	208 self._bakeRealmPages(record, pool, realm, all_factories)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	209 finally:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	210 page_count = record.current.baked_count[realm]
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	211 logger.info(format_timed(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	212 start_time,
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	213 "baked %d %s pages." %
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	214 (page_count, REALM_NAMES[realm].lower())))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	215
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	216 def _loadRealmPages(self, record, pool, factories):
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	217 logger.debug("Loading %d realm pages..." % len(factories))
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	218 with format_timed_scope(logger,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	219 "loaded %d pages" % len(factories),
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	220 level=logging.DEBUG, colored=False,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	221 timer_env=self.app.env,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	222 timer_category='LoadJob'):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	223 for fac in factories:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	224 job = BakeWorkerJob(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	225 JOB_LOAD,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	226 LoadJobPayload(fac))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	227 pool.queue.put_nowait(job)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	228
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	229 def _handler(res):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	230 # Create the record entry for this page.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	231 record_entry = BakeRecordEntry(res.source_name, res.path)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	232 record_entry.config = res.config
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	233 if res.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	234 record_entry.errors += res.errors
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	235 record.current.success = False
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	236 self._logErrors(res.path, res.errors)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	237 record.addEntry(record_entry)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	238
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	239 self._waitOnWorkerPool(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	240 pool,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	241 expected_result_count=len(factories),
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	242 result_handler=_handler)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	243
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	244 def _renderRealmPages(self, record, pool, factories):
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	245 logger.debug("Rendering %d realm pages..." % len(factories))
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	246 with format_timed_scope(logger,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	247 "prepared %d pages" % len(factories),
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	248 level=logging.DEBUG, colored=False,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	249 timer_env=self.app.env,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	250 timer_category='RenderFirstSubJob'):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	251 expected_result_count = 0
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	252 for fac in factories:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	253 record_entry = record.getCurrentEntry(fac.path)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	254 if record_entry.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	255 logger.debug("Ignoring %s because it had previous "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	256 "errors." % fac.ref_spec)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	257 continue
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	258
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	259 # Make sure the source and the route exist for this page,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	260 # otherwise we add errors to the record entry and we'll skip
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	261 # this page for the rest of the bake.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	262 source = self.app.getSource(fac.source.name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	263 if source is None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	264 record_entry.errors.append(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	265 "Can't get source for page: %s" % fac.ref_spec)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	266 logger.error(record_entry.errors[-1])
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	267 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	268
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	269 route = self.app.getRoute(fac.source.name, fac.metadata,
363 dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 338 diff changeset	270 skip_taxonomies=True)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	271 if route is None:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	272 record_entry.errors.append(
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	273 "Can't get route for page: %s" % fac.ref_spec)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	274 logger.error(record_entry.errors[-1])
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	275 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	276
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	277 # All good, queue the job.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	278 job = BakeWorkerJob(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	279 JOB_RENDER_FIRST,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	280 RenderFirstSubJobPayload(fac))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	281 pool.queue.put_nowait(job)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	282 expected_result_count += 1
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	283
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	284 def _handler(res):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	285 entry = record.getCurrentEntry(res.path)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	286 if res.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	287 entry.errors += res.errors
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	288 record.current.success = False
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	289 self._logErrors(res.path, res.errors)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	290
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	291 self._waitOnWorkerPool(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	292 pool,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	293 expected_result_count=expected_result_count,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	294 result_handler=_handler)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	295
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	296 def _bakeRealmPages(self, record, pool, realm, factories):
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	297 logger.debug("Baking %d realm pages..." % len(factories))
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	298 with format_timed_scope(logger,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	299 "baked %d pages" % len(factories),
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	300 level=logging.DEBUG, colored=False,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	301 timer_env=self.app.env,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	302 timer_category='BakeJob'):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	303 expected_result_count = 0
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	304 for fac in factories:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	305 if self._queueBakeJob(record, pool, fac):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	306 expected_result_count += 1
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	307
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	308 def _handler(res):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	309 entry = record.getCurrentEntry(res.path, res.taxonomy_info)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	310 entry.subs = res.sub_entries
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	311 if res.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	312 entry.errors += res.errors
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	313 self._logErrors(res.path, res.errors)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	314 if entry.has_any_error:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	315 record.current.success = False
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	316 if entry.was_any_sub_baked:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	317 record.current.baked_count[realm] += 1
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	318 record.dirty_source_names.add(entry.source_name)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	319
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	320 self._waitOnWorkerPool(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	321 pool,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	322 expected_result_count=expected_result_count,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	323 result_handler=_handler)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	324
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	325 def _bakeTaxonomies(self, record, pool):
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	326 logger.debug("Baking taxonomy pages...")
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	327 with format_timed_scope(logger, 'built taxonomy buckets',
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	328 level=logging.DEBUG, colored=False):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	329 buckets = self._buildTaxonomyBuckets(record)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	330
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	331 start_time = time.perf_counter()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	332 page_count = self._bakeTaxonomyBuckets(record, pool, buckets)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	333 logger.info(format_timed(start_time,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	334 "baked %d taxonomy pages." % page_count))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	335
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	336 def _buildTaxonomyBuckets(self, record):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	337 # Let's see all the taxonomy terms for which we must bake a
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	338 # listing page... first, pre-populate our big map of used terms.
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	339 # For each source name, we have a list of taxonomies, and for each
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	340 # taxonomies, a list of terms, some being 'dirty', some used last
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	341 # time, etc.
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	342 buckets = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	343 tax_names = [t.name for t in self.app.taxonomies]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	344 source_names = [s.name for s in self.app.sources]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	345 for sn in source_names:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	346 source_taxonomies = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	347 buckets[sn] = source_taxonomies
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	348 for tn in tax_names:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	349 source_taxonomies[tn] = _TaxonomyTermsInfo()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	350
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	351 # Now see which ones are 'dirty' based on our bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	352 logger.debug("Gathering dirty taxonomy terms")
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	353 for prev_entry, cur_entry in record.transitions.values():
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	354 # Re-bake all taxonomy pages that include new or changed
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	355 # pages.
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	356 if cur_entry and cur_entry.was_any_sub_baked:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	357 entries = [cur_entry]
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	358 if prev_entry:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	359 entries.append(prev_entry)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	360
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	361 for tax in self.app.taxonomies:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	362 changed_terms = set()
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	363 for e in entries:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	364 terms = e.config.get(tax.setting_name)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	365 if terms:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	366 if not tax.is_multiple:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	367 terms = [terms]
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	368 changed_terms \|= set(terms)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	369
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	370 if len(changed_terms) > 0:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	371 tt_info = buckets[cur_entry.source_name][tax.name]
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	372 tt_info.dirty_terms \|= changed_terms
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	373
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	374 # Remember all terms used.
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	375 for tax in self.app.taxonomies:
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	376 if cur_entry and not cur_entry.was_overriden:
324 65e6d72f3877 bake/serve: Fix how taxonomy index pages are setup and rendered. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	377 cur_terms = cur_entry.config.get(tax.setting_name)
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	378 if cur_terms:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	379 if not tax.is_multiple:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	380 cur_terms = [cur_terms]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	381 tt_info = buckets[cur_entry.source_name][tax.name]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	382 tt_info.all_terms \|= set(cur_terms)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	383
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	384 # Re-bake the combination pages for terms that are 'dirty'.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	385 known_combinations = set()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	386 logger.debug("Gathering dirty term combinations")
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	387 for prev_entry, cur_entry in record.transitions.values():
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	388 if not cur_entry:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	389 continue
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	390 used_taxonomy_terms = cur_entry.getAllUsedTaxonomyTerms()
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	391 for sn, tn, terms in used_taxonomy_terms:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	392 if isinstance(terms, tuple):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	393 known_combinations.add((sn, tn, terms))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	394 for sn, tn, terms in known_combinations:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	395 tt_info = buckets[sn][tn]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	396 tt_info.all_terms.add(terms)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	397 if not tt_info.dirty_terms.isdisjoint(set(terms)):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	398 tt_info.dirty_terms.add(terms)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	399
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	400 return buckets
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	401
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	402 def _bakeTaxonomyBuckets(self, record, pool, buckets):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	403 # Start baking those terms.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	404 expected_result_count = 0
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	405 for source_name, source_taxonomies in buckets.items():
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	406 for tax_name, tt_info in source_taxonomies.items():
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	407 terms = tt_info.dirty_terms
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	408 if len(terms) == 0:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	409 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	410
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	411 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	412 "Baking '%s' for source '%s': %s" %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	413 (tax_name, source_name, terms))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	414 tax = self.app.getTaxonomy(tax_name)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	415 source = self.app.getSource(source_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	416 tax_page_ref = tax.getPageRef(source)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	417 if not tax_page_ref.exists:
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	418 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	419 "No taxonomy page found at '%s', skipping." %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	420 tax.page_ref)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	421 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	422
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	423 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	424 "Using taxonomy page: %s:%s" %
369 4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	425 (tax_page_ref.source_name, tax_page_ref.rel_path))
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	426 fac = tax_page_ref.getFactory()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	427
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	428 for term in terms:
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	429 logger.debug(
369 4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	430 "Queuing: %s [%s=%s]" %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	431 (fac.ref_spec, tax_name, term))
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	432 tax_info = TaxonomyInfo(tax_name, source_name, term)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	433
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	434 cur_entry = BakeRecordEntry(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	435 fac.source.name, fac.path, tax_info)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	436 record.addEntry(cur_entry)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	437
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	438 if self._queueBakeJob(record, pool, fac, tax_info):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	439 expected_result_count += 1
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	440
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	441 def _handler(res):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	442 entry = record.getCurrentEntry(res.path, res.taxonomy_info)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	443 entry.subs = res.sub_entries
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	444 if res.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	445 entry.errors += res.errors
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	446 if entry.has_any_error:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	447 record.current.success = False
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	448
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	449 self._waitOnWorkerPool(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	450 pool,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	451 expected_result_count=expected_result_count,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	452 result_handler=_handler)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	453
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	454 # Now we create bake entries for all the terms that were not dirty.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	455 # This is because otherwise, on the next incremental bake, we wouldn't
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	456 # find any entry for those things, and figure that we need to delete
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	457 # their outputs.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	458 for prev_entry, cur_entry in record.transitions.values():
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	459 # Only consider taxonomy-related entries that don't have any
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	460 # current version.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	461 if (prev_entry and prev_entry.taxonomy_info and
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	462 not cur_entry):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	463 ti = prev_entry.taxonomy_info
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	464 tt_info = buckets[ti.source_name][ti.taxonomy_name]
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	465 if ti.term in tt_info.all_terms:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	466 logger.debug("Creating unbaked entry for taxonomy "
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	467 "term '%s:%s'." % (ti.taxonomy_name, ti.term))
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	468 record.collapseEntry(prev_entry)
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	469 else:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	470 logger.debug("Taxonomy term '%s:%s' isn't used anymore." %
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	471 (ti.taxonomy_name, ti.term))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	472
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	473 return expected_result_count
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	474
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	475 def _queueBakeJob(self, record, pool, fac, tax_info=None):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	476 # Get the previous (if any) and current entry for this page.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	477 pair = record.getPreviousAndCurrentEntries(fac.path, tax_info)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	478 assert pair is not None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	479 prev_entry, cur_entry = pair
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	480 assert cur_entry is not None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	481
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	482 # Ignore if there were errors in the previous passes.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	483 if cur_entry.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	484 logger.debug("Ignoring %s because it had previous "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	485 "errors." % fac.ref_spec)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	486 return False
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	487
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	488 # Build the route metadata and find the appropriate route.
430 21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	489 page = fac.buildPage()
21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	490 route_metadata = create_route_metadata(page)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	491 if tax_info is not None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	492 tax = self.app.getTaxonomy(tax_info.taxonomy_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	493 route = self.app.getTaxonomyRoute(tax_info.taxonomy_name,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	494 tax_info.source_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	495
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	496 slugified_term = route.slugifyTaxonomyTerm(tax_info.term)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	497 route_metadata[tax.term_name] = slugified_term
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	498 else:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	499 route = self.app.getRoute(fac.source.name, route_metadata,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	500 skip_taxonomies=True)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	501 assert route is not None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	502
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	503 # Figure out if this page is overriden by another previously
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	504 # baked page. This happens for example when the user has
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	505 # made a page that has the same page/URL as a theme page.
430 21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	506 uri = route.getUri(route_metadata)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	507 override_entry = record.getOverrideEntry(page.path, uri)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	508 if override_entry is not None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	509 override_source = self.app.getSource(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	510 override_entry.source_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	511 if override_source.realm == fac.source.realm:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	512 cur_entry.errors.append(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	513 "Page '%s' maps to URL '%s' but is overriden "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	514 "by page '%s'." %
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	515 (fac.ref_spec, uri, override_entry.path))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	516 logger.error(cur_entry.errors[-1])
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	517 cur_entry.flags \|= BakeRecordEntry.FLAG_OVERRIDEN
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	518 return False
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	519
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	520 job = BakeWorkerJob(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	521 JOB_BAKE,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	522 BakeJobPayload(fac, route_metadata, prev_entry,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	523 record.dirty_source_names,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	524 tax_info))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	525 pool.queue.put_nowait(job)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	526 return True
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	527
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	528 def _handleDeletetions(self, record):
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	529 logger.debug("Handling deletions...")
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	530 for path, reason in record.getDeletions():
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	531 logger.debug("Removing '%s': %s" % (path, reason))
146 0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	532 try:
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	533 os.remove(path)
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	534 logger.info('[delete] %s' % path)
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	535 except OSError:
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	536 # Not a big deal if that file had already been removed
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	537 # by the user.
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	538 pass
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	539
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	540 def _logErrors(self, path, errors):
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	541 rel_path = os.path.relpath(path, self.app.root_dir)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	542 logger.error("Errors found in %s:" % rel_path)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	543 for e in errors:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	544 logger.error(" " + e)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	545
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	546 def _createWorkerPool(self):
426 ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	547 import sys
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	548 from piecrust.baking.worker import BakeWorkerContext, worker_func
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	549
426 ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	550 main_module = sys.modules['__main__']
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	551 is_profiling = os.path.basename(main_module.__file__) in [
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	552 'profile.py', 'cProfile.py']
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	553
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	554 pool = _WorkerPool()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	555 for i in range(self.num_workers):
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	556 ctx = BakeWorkerContext(
425 afeebdd9f767 bake: Pass the sub-cache directory to the bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 421 diff changeset	557 self.app.root_dir, self.app.cache.base_dir, self.out_dir,
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	558 pool.queue, pool.results, pool.abort_event,
426 ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	559 force=self.force, debug=self.app.debug,
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	560 is_profiling=is_profiling)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	561 w = multiprocessing.Process(
421 4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	562 name='BakeWorker_%d' % i,
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	563 target=worker_func, args=(i, ctx))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	564 w.start()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	565 pool.workers.append(w)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	566 return pool
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	567
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	568 def _terminateWorkerPool(self, pool):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	569 pool.abort_event.set()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	570 for w in pool.workers:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	571 w.join()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	572
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	573 def _waitOnWorkerPool(self, pool,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	574 expected_result_count=-1, result_handler=None):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	575 assert result_handler is None or expected_result_count >= 0
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	576 abort_with_exception = None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	577 try:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	578 if result_handler is None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	579 pool.queue.join()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	580 else:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	581 got_count = 0
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	582 while got_count < expected_result_count:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	583 try:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	584 res = pool.results.get(True, 10)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	585 except queue.Empty:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	586 logger.error(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	587 "Got %d results, expected %d, and timed-out "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	588 "for 10 seconds. A worker might be stuck?" %
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	589 (got_count, expected_result_count))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	590 abort_with_exception = Exception("Worker time-out.")
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	591 break
159 232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	592
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	593 if isinstance(res, dict) and res.get('type') == 'error':
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	594 abort_with_exception = Exception(
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	595 'Worker critical error:\n' +
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	596 '\n'.join(res['messages']))
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	597 break
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	598
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	599 got_count += 1
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	600 result_handler(res)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	601 except KeyboardInterrupt as kiex:
159 232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	602 logger.warning("Bake aborted by user... "
232989a6df36 Add support for KeyboardInterrupt in bake process. Ludovic Chabant <ludovic@chabant.com> parents: 158 diff changeset	603 "waiting for workers to stop.")
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	604 abort_with_exception = kiex
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	605
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	606 if abort_with_exception:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	607 pool.abort_event.set()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	608 for w in pool.workers:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	609 w.join(2)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	610 raise abort_with_exception
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	611
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	612
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	613 class _WorkerPool(object):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	614 def __init__(self):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	615 self.queue = multiprocessing.JoinableQueue()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	616 self.results = multiprocessing.Queue()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	617 self.abort_event = multiprocessing.Event()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	618 self.workers = []
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	619
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	620
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	621 class _TaxonomyTermsInfo(object):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	622 def __init__(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	623 self.dirty_terms = set()
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	624 self.all_terms = set()
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	625
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	626 def __str__(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	627 return 'dirty:%s, all:%s' % (self.dirty_terms, self.all_terms)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	628
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	629 def __repr__(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	630 return 'dirty:%s, all:%s' % (self.dirty_terms, self.all_terms)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	631

Mercurial > piecrust2

annotate piecrust/baking/baker.py @ 440:32c7c2d219d2