piecrust2: piecrust/baking/baker.py annotate

annotate piecrust/baking/baker.py @ 702:c62d83e17abf

bake: Some more optimizations. Write the output files in a background thread to get away from the GIL.

author	Ludovic Chabant <ludovic@chabant.com>
date	Sat, 16 Apr 2016 22:50:07 -0700
parents	9ae9390192da
children	5f552aedd918

rev	line source
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import time
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os.path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import hashlib
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import logging
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	5 from piecrust.baking.records import (
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	6 BakeRecordEntry, TransitionalBakeRecord, TaxonomyInfo)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	7 from piecrust.baking.worker import (
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	8 save_factory,
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	9 JOB_LOAD, JOB_RENDER_FIRST, JOB_BAKE)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	10 from piecrust.chefutil import (
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	11 format_timed_scope, format_timed)
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	12 from piecrust.environment import ExecutionStats
430 21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	13 from piecrust.routing import create_route_metadata
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	14 from piecrust.sources.base import (
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	15 REALM_NAMES, REALM_USER, REALM_THEME)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21 class Baker(object):
466 456db44dcc53 bake: Pass the config variants and values from the CLI to the baker. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	22 def __init__(self, app, out_dir, force=False,
456db44dcc53 bake: Pass the config variants and values from the CLI to the baker. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	23 applied_config_variant=None,
456db44dcc53 bake: Pass the config variants and values from the CLI to the baker. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	24 applied_config_values=None):
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	25 assert app and out_dir
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26 self.app = app
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	27 self.out_dir = out_dir
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 self.force = force
466 456db44dcc53 bake: Pass the config variants and values from the CLI to the baker. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	29 self.applied_config_variant = applied_config_variant
456db44dcc53 bake: Pass the config variants and values from the CLI to the baker. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	30 self.applied_config_values = applied_config_values
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32 # Remember what taxonomy pages we should skip
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 # (we'll bake them repeatedly later with each taxonomy term)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34 self.taxonomy_pages = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35 logger.debug("Gathering taxonomy page paths:")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 for tax in self.app.taxonomies:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37 for src in self.app.sources:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	38 tax_page_ref = tax.getPageRef(src)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	39 for path in tax_page_ref.possible_paths:
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	40 self.taxonomy_pages.append(path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41 logger.debug(" - %s" % path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	43 # Register some timers.
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	44 self.app.env.registerTimer('LoadJob', raise_if_registered=False)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	45 self.app.env.registerTimer('RenderFirstSubJob',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	46 raise_if_registered=False)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	47 self.app.env.registerTimer('BakeJob', raise_if_registered=False)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	48
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 def bake(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50 logger.debug(" Bake Output: %s" % self.out_dir)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51 logger.debug(" Root URL: %s" % self.app.config.get('site/root'))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53 # Get into bake mode.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	54 start_time = time.perf_counter()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	55 self.app.config.set('baker/is_baking', True)
32 43091c9837bf Fix problems with asset URLs. Ludovic Chabant <ludovic@chabant.com> parents: 25 diff changeset	56 self.app.env.base_asset_url_format = '%uri%'
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	57
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58 # Make sure the output directory exists.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 if not os.path.isdir(self.out_dir):
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	60 os.makedirs(self.out_dir, 0o755)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	61
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62 # Load/create the bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63 record = TransitionalBakeRecord()
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 42 diff changeset	64 record_cache = self.app.cache.getCache('baker')
333 91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	65 record_id = hashlib.md5(self.out_dir.encode('utf8')).hexdigest()
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	66 record_name = record_id + '.record'
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	67 previous_record_path = None
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 if not self.force and record_cache.has(record_name):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	69 with format_timed_scope(logger, "loaded previous bake record",
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	70 level=logging.DEBUG, colored=False):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	71 previous_record_path = record_cache.getCachePath(record_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	72 record.loadPrevious(previous_record_path)
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	73 record.current.success = True
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	74
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	75 # Figure out if we need to clean the cache because important things
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	76 # have changed.
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	77 is_cache_valid = self._handleCacheValidity(record)
8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	78 if not is_cache_valid:
8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	79 previous_record_path = None
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	80
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	81 # Pre-create all caches.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	82 for cache_name in ['app', 'baker', 'pages', 'renders']:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	83 self.app.cache.getCache(cache_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	84
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	85 # Gather all sources by realm -- we're going to bake each realm
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	86 # separately so we can handle "overriding" (i.e. one realm overrides
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	87 # another realm's pages, like the user realm overriding the theme
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	88 # realm).
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	89 sources_by_realm = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	90 for source in self.app.sources:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	91 srclist = sources_by_realm.setdefault(source.realm, [])
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	92 srclist.append(source)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	93
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	94 # Create the worker processes.
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	95 pool = self._createWorkerPool(previous_record_path)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	96
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	97 # Bake the realms.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	98 realm_list = [REALM_USER, REALM_THEME]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	99 for realm in realm_list:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	100 srclist = sources_by_realm.get(realm)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	101 if srclist is not None:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	102 self._bakeRealm(record, pool, realm, srclist)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	103
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	104 # Bake taxonomies.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	105 self._bakeTaxonomies(record, pool)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	106
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	107 # All done with the workers. Close the pool and get reports.
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	108 reports = pool.close()
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	109 total_stats = ExecutionStats()
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	110 record.current.stats['_Total'] = total_stats
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	111 for i in range(len(reports)):
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	112 worker_stats = reports[i]['data']
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	113 if worker_stats is not None:
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	114 worker_name = 'BakeWorker_%d' % i
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	115 record.current.stats[worker_name] = worker_stats
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	116 total_stats.mergeStats(worker_stats)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	117
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	118 # Delete files from the output.
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	119 self._handleDeletetions(record)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	120
333 91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	121 # Backup previous records.
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	122 for i in range(8, -1, -1):
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	123 suffix = '' if i == 0 else '.%d' % i
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	124 record_path = record_cache.getCachePath(
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	125 '%s%s.record' % (record_id, suffix))
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	126 if os.path.exists(record_path):
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	127 record_path_next = record_cache.getCachePath(
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	128 '%s.%s.record' % (record_id, i + 1))
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	129 if os.path.exists(record_path_next):
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	130 os.remove(record_path_next)
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	131 os.rename(record_path, record_path_next)
91b07f9efdc1 bake: Use a rotating bake record. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	132
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	133 # Save the bake record.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	134 with format_timed_scope(logger, "saved bake record.",
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	135 level=logging.DEBUG, colored=False):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	136 record.current.bake_time = time.time()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	137 record.current.out_dir = self.out_dir
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	138 record.saveCurrent(record_cache.getCachePath(record_name))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	139
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	140 # All done.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	141 self.app.config.set('baker/is_baking', False)
151 fd146f54bdaa Forgot this wasn't C++. Ludovic Chabant <ludovic@chabant.com> parents: 150 diff changeset	142 logger.debug(format_timed(start_time, 'done baking'))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	143
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	144 return record.detach()
1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	145
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	146 def _handleCacheValidity(self, record):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	147 start_time = time.perf_counter()
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	148
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	149 reason = None
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	150 if self.force:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	151 reason = "ordered to"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	152 elif not self.app.config.get('__cache_valid'):
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	153 # The configuration file was changed, or we're running a new
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	154 # version of the app.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	155 reason = "not valid anymore"
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	156 elif (not record.previous.bake_time or
e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	157 not record.previous.hasLatestVersion()):
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	158 # We have no valid previous bake record.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	159 reason = "need bake record regeneration"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	160 else:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	161 # Check if any template has changed since the last bake. Since
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	162 # there could be some advanced conditional logic going on, we'd
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	163 # better just force a bake from scratch if that's the case.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	164 max_time = 0
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	165 for d in self.app.templates_dirs:
42 9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	166 for dpath, _, filenames in os.walk(d):
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	167 for fn in filenames:
42 9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	168 full_fn = os.path.join(dpath, fn)
9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	169 max_time = max(max_time, os.path.getmtime(full_fn))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	170 if max_time >= record.previous.bake_time:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	171 reason = "templates modified"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	172
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	173 if reason is not None:
85 3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests. Ludovic Chabant <ludovic@chabant.com> parents: 50 diff changeset	174 # We have to bake everything from scratch.
371 c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	175 self.app.cache.clearCaches(except_names=['app'])
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 42 diff changeset	176 self.force = True
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	177 record.incremental_count = 0
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 91 diff changeset	178 record.clearPrevious()
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	179 logger.info(format_timed(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	180 start_time,
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	181 "cleaned cache (reason: %s)" % reason))
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	182 return False
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	183 else:
91 e88e330eb8dc Improvements to incremental baking and cache invalidating. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	184 record.incremental_count += 1
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	185 logger.debug(format_timed(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	186 start_time, "cache is assumed valid",
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	187 colored=False))
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	188 return True
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	189
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	190 def _bakeRealm(self, record, pool, realm, srclist):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	191 start_time = time.perf_counter()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	192 try:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	193 record.current.baked_count[realm] = 0
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	194 record.current.total_baked_count[realm] = 0
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	195
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	196 all_factories = []
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	197 for source in srclist:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	198 factories = source.getPageFactories()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	199 all_factories += [f for f in factories
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	200 if f.path not in self.taxonomy_pages]
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	201
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	202 self._loadRealmPages(record, pool, all_factories)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	203 self._renderRealmPages(record, pool, all_factories)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	204 self._bakeRealmPages(record, pool, realm, all_factories)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	205 finally:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	206 page_count = record.current.baked_count[realm]
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	207 total_page_count = record.current.total_baked_count[realm]
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	208 logger.info(format_timed(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	209 start_time,
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	210 "baked %d %s pages (%d total)." %
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	211 (page_count, REALM_NAMES[realm].lower(),
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	212 total_page_count)))
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	213
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	214 def _loadRealmPages(self, record, pool, factories):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	215 def _handler(res):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	216 # Create the record entry for this page.
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	217 # This will also update the `dirty_source_names` for the record
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	218 # as we add page files whose last modification times are later
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	219 # than the last bake.
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	220 record_entry = BakeRecordEntry(res['source_name'], res['path'])
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	221 record_entry.config = res['config']
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	222 if res['errors']:
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	223 record_entry.errors += res['errors']
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	224 record.current.success = False
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	225 self._logErrors(res['path'], res['errors'])
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	226 record.addEntry(record_entry)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	227
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	228 logger.debug("Loading %d realm pages..." % len(factories))
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	229 with format_timed_scope(logger,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	230 "loaded %d pages" % len(factories),
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	231 level=logging.DEBUG, colored=False,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	232 timer_env=self.app.env,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	233 timer_category='LoadJob'):
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	234 jobs = []
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	235 for fac in factories:
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	236 job = {
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	237 'type': JOB_LOAD,
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	238 'job': save_factory(fac)}
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	239 jobs.append(job)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	240 ar = pool.queueJobs(jobs, handler=_handler)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	241 ar.wait()
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	242
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	243 def _renderRealmPages(self, record, pool, factories):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	244 def _handler(res):
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	245 entry = record.getCurrentEntry(res['path'])
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	246 if res['errors']:
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	247 entry.errors += res['errors']
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	248 record.current.success = False
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	249 self._logErrors(res['path'], res['errors'])
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	250
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	251 logger.debug("Rendering %d realm pages..." % len(factories))
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	252 with format_timed_scope(logger,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	253 "prepared %d pages" % len(factories),
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	254 level=logging.DEBUG, colored=False,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	255 timer_env=self.app.env,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	256 timer_category='RenderFirstSubJob'):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	257 jobs = []
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	258 for fac in factories:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	259 record_entry = record.getCurrentEntry(fac.path)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	260 if record_entry.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	261 logger.debug("Ignoring %s because it had previous "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	262 "errors." % fac.ref_spec)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	263 continue
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	264
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	265 # Make sure the source and the route exist for this page,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	266 # otherwise we add errors to the record entry and we'll skip
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	267 # this page for the rest of the bake.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	268 source = self.app.getSource(fac.source.name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	269 if source is None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	270 record_entry.errors.append(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	271 "Can't get source for page: %s" % fac.ref_spec)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	272 logger.error(record_entry.errors[-1])
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	273 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	274
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	275 route = self.app.getRoute(fac.source.name, fac.metadata,
363 dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 338 diff changeset	276 skip_taxonomies=True)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	277 if route is None:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	278 record_entry.errors.append(
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	279 "Can't get route for page: %s" % fac.ref_spec)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	280 logger.error(record_entry.errors[-1])
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	281 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	282
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	283 # All good, queue the job.
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	284 job = {
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	285 'type': JOB_RENDER_FIRST,
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	286 'job': save_factory(fac)}
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	287 jobs.append(job)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	288
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	289 ar = pool.queueJobs(jobs, handler=_handler)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	290 ar.wait()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	291
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	292 def _bakeRealmPages(self, record, pool, realm, factories):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	293 def _handler(res):
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	294 entry = record.getCurrentEntry(res['path'], res['taxonomy_info'])
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	295 entry.subs = res['sub_entries']
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	296 if res['errors']:
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	297 entry.errors += res['errors']
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	298 self._logErrors(res['path'], res['errors'])
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	299 if entry.has_any_error:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	300 record.current.success = False
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	301 if entry.subs and entry.was_any_sub_baked:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	302 record.current.baked_count[realm] += 1
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	303 record.current.total_baked_count[realm] += len(entry.subs)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	304
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	305 logger.debug("Baking %d realm pages..." % len(factories))
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	306 with format_timed_scope(logger,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	307 "baked %d pages" % len(factories),
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	308 level=logging.DEBUG, colored=False,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	309 timer_env=self.app.env,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	310 timer_category='BakeJob'):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	311 jobs = []
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	312 for fac in factories:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	313 job = self._makeBakeJob(record, fac)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	314 if job is not None:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	315 jobs.append(job)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	316
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	317 ar = pool.queueJobs(jobs, handler=_handler)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	318 ar.wait()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	319
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	320 def _bakeTaxonomies(self, record, pool):
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	321 logger.debug("Baking taxonomy pages...")
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	322 with format_timed_scope(logger, 'built taxonomy buckets',
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	323 level=logging.DEBUG, colored=False):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	324 buckets = self._buildTaxonomyBuckets(record)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	325
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	326 start_time = time.perf_counter()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	327 page_count = self._bakeTaxonomyBuckets(record, pool, buckets)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	328 logger.info(format_timed(start_time,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	329 "baked %d taxonomy pages." % page_count))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	330
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	331 def _buildTaxonomyBuckets(self, record):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	332 # Let's see all the taxonomy terms for which we must bake a
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	333 # listing page... first, pre-populate our big map of used terms.
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	334 # For each source name, we have a list of taxonomies, and for each
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	335 # taxonomies, a list of terms, some being 'dirty', some used last
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	336 # time, etc.
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	337 buckets = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	338 tax_names = [t.name for t in self.app.taxonomies]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	339 source_names = [s.name for s in self.app.sources]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	340 for sn in source_names:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	341 source_taxonomies = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	342 buckets[sn] = source_taxonomies
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	343 for tn in tax_names:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	344 source_taxonomies[tn] = _TaxonomyTermsInfo()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	345
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	346 # Now see which ones are 'dirty' based on our bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	347 logger.debug("Gathering dirty taxonomy terms")
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	348 for prev_entry, cur_entry in record.transitions.values():
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	349 # Re-bake all taxonomy pages that include new or changed
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	350 # pages.
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	351 if cur_entry and cur_entry.was_any_sub_baked:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	352 entries = [cur_entry]
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	353 if prev_entry:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	354 entries.append(prev_entry)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	355
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	356 for tax in self.app.taxonomies:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	357 changed_terms = set()
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	358 for e in entries:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	359 terms = e.config.get(tax.setting_name)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	360 if terms:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	361 if not tax.is_multiple:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	362 terms = [terms]
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	363 changed_terms \|= set(terms)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	364
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	365 if len(changed_terms) > 0:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	366 tt_info = buckets[cur_entry.source_name][tax.name]
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	367 tt_info.dirty_terms \|= changed_terms
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	368
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	369 # Remember all terms used.
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	370 for tax in self.app.taxonomies:
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	371 if cur_entry and not cur_entry.was_overriden:
324 65e6d72f3877 bake/serve: Fix how taxonomy index pages are setup and rendered. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	372 cur_terms = cur_entry.config.get(tax.setting_name)
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	373 if cur_terms:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	374 if not tax.is_multiple:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	375 cur_terms = [cur_terms]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	376 tt_info = buckets[cur_entry.source_name][tax.name]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	377 tt_info.all_terms \|= set(cur_terms)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	378
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	379 # Re-bake the combination pages for terms that are 'dirty'.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	380 known_combinations = set()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	381 logger.debug("Gathering dirty term combinations")
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	382 for prev_entry, cur_entry in record.transitions.values():
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	383 if not cur_entry:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	384 continue
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	385 used_taxonomy_terms = cur_entry.getAllUsedTaxonomyTerms()
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	386 for sn, tn, terms in used_taxonomy_terms:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	387 if isinstance(terms, tuple):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	388 known_combinations.add((sn, tn, terms))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	389 for sn, tn, terms in known_combinations:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	390 tt_info = buckets[sn][tn]
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	391 tt_info.all_terms.add(terms)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	392 if not tt_info.dirty_terms.isdisjoint(set(terms)):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	393 tt_info.dirty_terms.add(terms)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	394
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	395 return buckets
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	396
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	397 def _bakeTaxonomyBuckets(self, record, pool, buckets):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	398 def _handler(res):
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	399 entry = record.getCurrentEntry(res['path'], res['taxonomy_info'])
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	400 entry.subs = res['sub_entries']
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	401 if res['errors']:
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	402 entry.errors += res['errors']
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	403 if entry.has_any_error:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	404 record.current.success = False
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	405
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	406 # Start baking those terms.
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	407 jobs = []
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	408 for source_name, source_taxonomies in buckets.items():
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	409 for tax_name, tt_info in source_taxonomies.items():
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	410 terms = tt_info.dirty_terms
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	411 if len(terms) == 0:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	412 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	413
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	414 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	415 "Baking '%s' for source '%s': %s" %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	416 (tax_name, source_name, terms))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	417 tax = self.app.getTaxonomy(tax_name)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	418 source = self.app.getSource(source_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	419 tax_page_ref = tax.getPageRef(source)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	420 if not tax_page_ref.exists:
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	421 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	422 "No taxonomy page found at '%s', skipping." %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	423 tax.page_ref)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	424 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	425
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	426 logger.debug(
1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	427 "Using taxonomy page: %s:%s" %
369 4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	428 (tax_page_ref.source_name, tax_page_ref.rel_path))
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	429 fac = tax_page_ref.getFactory()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	430
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	431 for term in terms:
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	432 logger.debug(
369 4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	433 "Queuing: %s [%s=%s]" %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	434 (fac.ref_spec, tax_name, term))
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	435 tax_info = TaxonomyInfo(tax_name, source_name, term)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	436
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	437 cur_entry = BakeRecordEntry(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	438 fac.source.name, fac.path, tax_info)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	439 record.addEntry(cur_entry)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	440
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	441 job = self._makeBakeJob(record, fac, tax_info)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	442 if job is not None:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	443 jobs.append(job)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	444
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	445 ar = pool.queueJobs(jobs, handler=_handler)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	446 ar.wait()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	447
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	448 # Now we create bake entries for all the terms that were not dirty.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	449 # This is because otherwise, on the next incremental bake, we wouldn't
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	450 # find any entry for those things, and figure that we need to delete
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	451 # their outputs.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	452 for prev_entry, cur_entry in record.transitions.values():
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	453 # Only consider taxonomy-related entries that don't have any
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	454 # current version.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	455 if (prev_entry and prev_entry.taxonomy_info and
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	456 not cur_entry):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	457 ti = prev_entry.taxonomy_info
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	458 tt_info = buckets[ti.source_name][ti.taxonomy_name]
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	459 if ti.term in tt_info.all_terms:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	460 logger.debug("Creating unbaked entry for taxonomy "
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	461 "term '%s:%s'." % (ti.taxonomy_name, ti.term))
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	462 record.collapseEntry(prev_entry)
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	463 else:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	464 logger.debug("Taxonomy term '%s:%s' isn't used anymore." %
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	465 (ti.taxonomy_name, ti.term))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	466
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	467 return len(jobs)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	468
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	469 def _makeBakeJob(self, record, fac, tax_info=None):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	470 # Get the previous (if any) and current entry for this page.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	471 pair = record.getPreviousAndCurrentEntries(fac.path, tax_info)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	472 assert pair is not None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	473 prev_entry, cur_entry = pair
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	474 assert cur_entry is not None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	475
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	476 # Ignore if there were errors in the previous passes.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	477 if cur_entry.errors:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	478 logger.debug("Ignoring %s because it had previous "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	479 "errors." % fac.ref_spec)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	480 return None
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	481
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	482 # Build the route metadata and find the appropriate route.
430 21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	483 page = fac.buildPage()
21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	484 route_metadata = create_route_metadata(page)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	485 if tax_info is not None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	486 tax = self.app.getTaxonomy(tax_info.taxonomy_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	487 route = self.app.getTaxonomyRoute(tax_info.taxonomy_name,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	488 tax_info.source_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	489
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	490 slugified_term = route.slugifyTaxonomyTerm(tax_info.term)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	491 route_metadata[tax.term_name] = slugified_term
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	492 else:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	493 route = self.app.getRoute(fac.source.name, route_metadata,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	494 skip_taxonomies=True)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	495 assert route is not None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	496
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	497 # Figure out if this page is overriden by another previously
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	498 # baked page. This happens for example when the user has
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	499 # made a page that has the same page/URL as a theme page.
430 21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	500 uri = route.getUri(route_metadata)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	501 override_entry = record.getOverrideEntry(page.path, uri)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	502 if override_entry is not None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	503 override_source = self.app.getSource(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	504 override_entry.source_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	505 if override_source.realm == fac.source.realm:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	506 cur_entry.errors.append(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	507 "Page '%s' maps to URL '%s' but is overriden "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	508 "by page '%s'." %
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	509 (fac.ref_spec, uri, override_entry.path))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	510 logger.error(cur_entry.errors[-1])
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	511 cur_entry.flags \|= BakeRecordEntry.FLAG_OVERRIDEN
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	512 return None
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	513
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	514 job = {
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	515 'type': JOB_BAKE,
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	516 'job': {
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	517 'factory_info': save_factory(fac),
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	518 'taxonomy_info': tax_info,
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	519 'route_metadata': route_metadata,
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	520 'dirty_source_names': record.dirty_source_names
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	521 }
838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	522 }
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	523 return job
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 333 diff changeset	524
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	525 def _handleDeletetions(self, record):
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	526 logger.debug("Handling deletions...")
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	527 for path, reason in record.getDeletions():
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	528 logger.debug("Removing '%s': %s" % (path, reason))
146 0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	529 try:
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	530 os.remove(path)
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	531 logger.info('[delete] %s' % path)
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	532 except OSError:
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	533 # Not a big deal if that file had already been removed
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	534 # by the user.
0609739169bd Don't fail if trying to clean up a file that has already been deleted. Ludovic Chabant <ludovic@chabant.com> parents: 133 diff changeset	535 pass
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	536
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	537 def _logErrors(self, path, errors):
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	538 rel_path = os.path.relpath(path, self.app.root_dir)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	539 logger.error("Errors found in %s:" % rel_path)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	540 for e in errors:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	541 logger.error(" " + e)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	542
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	543 def _createWorkerPool(self, previous_record_path):
666 81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	544 from piecrust.app import PieCrustFactory
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	545 from piecrust.workerpool import WorkerPool
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	546 from piecrust.baking.worker import BakeWorkerContext, BakeWorker
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	547
666 81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	548 appfactory = PieCrustFactory(
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	549 self.app.root_dir,
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	550 cache=self.app.cache.enabled,
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	551 cache_key=self.app.cache_key,
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	552 config_variant=self.applied_config_variant,
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	553 config_values=self.applied_config_values,
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	554 debug=self.app.debug,
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	555 theme_site=self.app.theme_site)
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	556
456 5e902e228053 bake: Correctly use the `num_worers` setting. Ludovic Chabant <ludovic@chabant.com> parents: 453 diff changeset	557 worker_count = self.app.config.get('baker/workers')
462 04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 456 diff changeset	558 batch_size = self.app.config.get('baker/batch_size')
456 5e902e228053 bake: Correctly use the `num_worers` setting. Ludovic Chabant <ludovic@chabant.com> parents: 453 diff changeset	559
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	560 ctx = BakeWorkerContext(
666 81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	561 appfactory,
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	562 self.out_dir,
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	563 force=self.force,
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	564 previous_record_path=previous_record_path)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	565 pool = WorkerPool(
456 5e902e228053 bake: Correctly use the `num_worers` setting. Ludovic Chabant <ludovic@chabant.com> parents: 453 diff changeset	566 worker_count=worker_count,
462 04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 456 diff changeset	567 batch_size=batch_size,
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	568 worker_class=BakeWorker,
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	569 initargs=(ctx,))
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	570 return pool
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	571
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	572
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	573 class _TaxonomyTermsInfo(object):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	574 def __init__(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	575 self.dirty_terms = set()
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	576 self.all_terms = set()
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	577
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	578 def __str__(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	579 return 'dirty:%s, all:%s' % (self.dirty_terms, self.all_terms)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	580
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	581 def __repr__(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	582 return 'dirty:%s, all:%s' % (self.dirty_terms, self.all_terms)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	583

Mercurial > piecrust2

annotate piecrust/baking/baker.py @ 702:c62d83e17abf