piecrust2: piecrust/baking/baker.py annotate

annotate piecrust/baking/baker.py @ 1014:071f30aa04bb

bake: Do template caching in a background job if possible.

author	Ludovic Chabant <ludovic@chabant.com>
date	Tue, 28 Nov 2017 21:28:15 -0800
parents	c4cf3cfe2726
children	fa489c5e829e

rev	line source
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import time
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os.path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import hashlib
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import logging
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	5 from piecrust.chefutil import (
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	6 format_timed_scope, format_timed)
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	7 from piecrust.environment import ExecutionStats
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	8 from piecrust.pipelines.base import (
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	9 PipelineJobCreateContext, PipelineJobResultHandleContext,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	10 PipelineJobValidateContext, PipelineManager,
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	11 get_pipeline_name_for_source)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	12 from piecrust.pipelines.records import (
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	13 MultiRecordHistory, MultiRecord, RecordEntry,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	14 load_records)
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	15 from piecrust.sources.base import REALM_USER, REALM_THEME, REALM_NAMES
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	21 def get_bake_records_path(app, out_dir, *, suffix=''):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	22 records_cache = app.cache.getCache('baker')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	23 records_id = hashlib.md5(out_dir.encode('utf8')).hexdigest()
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	24 records_name = '%s%s.records' % (records_id, suffix)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	25 return records_cache.getCachePath(records_name)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	26
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	27
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 class Baker(object):
918 7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 899 diff changeset	29 def __init__(self, appfactory, app, out_dir, *,
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 899 diff changeset	30 force=False,
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 899 diff changeset	31 allowed_pipelines=None,
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 899 diff changeset	32 forbidden_pipelines=None,
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 899 diff changeset	33 allowed_sources=None,
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 899 diff changeset	34 rotate_bake_records=True):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	35 self.appfactory = appfactory
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 self.app = app
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	37 self.out_dir = out_dir
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	38 self.force = force
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	39 self.allowed_pipelines = allowed_pipelines
868 8d25f76fce98 bake: Add ability to specify pipelines to exclude during the bake. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	40 self.forbidden_pipelines = forbidden_pipelines
918 7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 899 diff changeset	41 self.allowed_sources = allowed_sources
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 899 diff changeset	42 self.rotate_bake_records = rotate_bake_records
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	43
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44 def bake(self):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	45 start_time = time.perf_counter()
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	46
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	47 # Setup baker.
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	48 logger.debug(" Bake Output: %s" % self.out_dir)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 logger.debug(" Root URL: %s" % self.app.config.get('site/root'))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51 # Get into bake mode.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52 self.app.config.set('baker/is_baking', True)
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	53 self.app.config.set('site/asset_url_format', '%page_uri%/%filename%')
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54
876 d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	55 stats = self.app.env.stats
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	56 stats.registerTimer('LoadSourceContents', raise_if_registered=False)
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1007 diff changeset	57 stats.registerTimer('CacheTemplates', raise_if_registered=False)
876 d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	58
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 # Make sure the output directory exists.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	60 if not os.path.isdir(self.out_dir):
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	61 os.makedirs(self.out_dir, 0o755)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	63 # Load/create the bake records.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	64 records_path = get_bake_records_path(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	65 self.app, self.out_dir)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	66 if not self.force and os.path.isfile(records_path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	67 with format_timed_scope(logger, "loaded previous bake records",
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	68 level=logging.DEBUG, colored=False):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	69 previous_records = load_records(records_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	70 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	71 previous_records = MultiRecord()
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	72 current_records = MultiRecord()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	73
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	74 # Figure out if we need to clean the cache because important things
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	75 # have changed.
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	76 is_cache_valid = self._handleCacheValidity(previous_records,
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	77 current_records)
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	78 if not is_cache_valid:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	79 previous_records = MultiRecord()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	80
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	81 # Create the bake records history which tracks what's up-to-date
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	82 # or not since last time we baked to the given output folder.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	83 record_histories = MultiRecordHistory(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	84 previous_records, current_records)
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	85
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	86 # Pre-create all caches.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	87 for cache_name in ['app', 'baker', 'pages', 'renders']:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	88 self.app.cache.getCache(cache_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	89
972 bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	90 # Create the pipelines.
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	91 ppmngr = self._createPipelineManager(record_histories)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	92
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	93 # Create the worker processes.
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	94 pool_userdata = _PoolUserData(self, ppmngr)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	95 pool = self._createWorkerPool(records_path, pool_userdata)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	96
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	97 # Done with all the setup, let's start the actual work.
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	98 logger.info(format_timed(start_time, "setup baker"))
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	99
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1007 diff changeset	100 # Load all sources, pre-cache templates.
1014 071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	101 load_start_time = time.perf_counter()
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	102 self._startPopulateTemplateCaches(pool)
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	103 self._loadSources(ppmngr)
1014 071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	104 self._endPopulateTemplateCache(pool)
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	105 logger.info(format_timed(load_start_time, "loaded site content"))
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	106
972 bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	107 # Bake the realms.
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	108 self._bakeRealms(pool, ppmngr, record_histories)
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	109
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	110 # Handle deletions, collapse records, etc.
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	111 ppmngr.postJobRun()
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	112 ppmngr.deleteStaleOutputs()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	113 ppmngr.collapseRecords()
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	114
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	115 # All done with the workers. Close the pool and get reports.
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	116 pool_stats = pool.close()
972 bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	117 current_records.stats = _merge_execution_stats(stats, *pool_stats)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	118
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	119 # Shutdown the pipelines.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	120 ppmngr.shutdownPipelines()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	121
972 bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	122 # Backup previous records, save the current ones.
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	123 current_records.bake_time = time.time()
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	124 current_records.out_dir = self.out_dir
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	125 _save_bake_records(current_records, records_path,
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	126 rotate_previous=self.rotate_bake_records)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	127
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	128 # All done.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	129 self.app.config.set('baker/is_baking', False)
151 fd146f54bdaa Forgot this wasn't C++. Ludovic Chabant <ludovic@chabant.com> parents: 150 diff changeset	130 logger.debug(format_timed(start_time, 'done baking'))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	131
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	132 return current_records
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	133
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	134 def _handleCacheValidity(self, previous_records, current_records):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	135 start_time = time.perf_counter()
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	136
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	137 reason = None
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	138 if self.force:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	139 reason = "ordered to"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	140 elif not self.app.config.get('__cache_valid'):
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	141 # The configuration file was changed, or we're running a new
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	142 # version of the app.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	143 reason = "not valid anymore"
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	144 elif previous_records.invalidated:
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	145 # We have no valid previous bake records.
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	146 reason = "need bake records regeneration"
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	147 else:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	148 # Check if any template has changed since the last bake. Since
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	149 # there could be some advanced conditional logic going on, we'd
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	150 # better just force a bake from scratch if that's the case.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	151 max_time = 0
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	152 for d in self.app.templates_dirs:
42 9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	153 for dpath, _, filenames in os.walk(d):
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	154 for fn in filenames:
42 9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	155 full_fn = os.path.join(dpath, fn)
9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	156 max_time = max(max_time, os.path.getmtime(full_fn))
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	157 if max_time >= previous_records.bake_time:
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	158 reason = "templates modified"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	159
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	160 if reason is not None:
85 3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests. Ludovic Chabant <ludovic@chabant.com> parents: 50 diff changeset	161 # We have to bake everything from scratch.
707 5f552aedd918 bake: Don't clean the `baker` cache on a force bake. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	162 self.app.cache.clearCaches(except_names=['app', 'baker'])
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 42 diff changeset	163 self.force = True
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	164 current_records.incremental_count = 0
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	165 previous_records = MultiRecord()
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	166 logger.debug(format_timed(
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	167 start_time, "cleaned cache (reason: %s)" % reason,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	168 colored=False))
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	169 return False
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	170 else:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	171 current_records.incremental_count += 1
158 1187739e5a19 Fix some indentation and line lengths. Ludovic Chabant <ludovic@chabant.com> parents: 151 diff changeset	172 logger.debug(format_timed(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	173 start_time, "cache is assumed valid", colored=False))
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	174 return True
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	175
972 bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	176 def _createPipelineManager(self, record_histories):
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	177 # Gather all sources by realm -- we're going to bake each realm
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	178 # separately so we can handle "overriding" (i.e. one realm overrides
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	179 # another realm's pages, like the user realm overriding the theme
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	180 # realm).
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	181 #
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	182 # Also, create and initialize each pipeline for each source.
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	183 has_any_pp = False
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	184 ppmngr = PipelineManager(
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	185 self.app, self.out_dir,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	186 record_histories=record_histories)
972 bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	187 ok_pp = self.allowed_pipelines
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	188 nok_pp = self.forbidden_pipelines
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	189 ok_src = self.allowed_sources
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	190 for source in self.app.sources:
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	191 if ok_src is not None and source.name not in ok_src:
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	192 continue
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	193
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	194 pname = get_pipeline_name_for_source(source)
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	195 if ok_pp is not None and pname not in ok_pp:
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	196 continue
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	197 if nok_pp is not None and pname in nok_pp:
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	198 continue
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	199
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	200 ppinfo = ppmngr.createPipeline(source)
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	201 logger.debug(
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	202 "Created pipeline '%s' for source: %s" %
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	203 (ppinfo.pipeline.PIPELINE_NAME, source.name))
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	204 has_any_pp = True
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	205 if not has_any_pp:
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	206 raise Exception("The website has no content sources, or the bake "
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	207 "command was invoked with all pipelines filtered "
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	208 "out. There's nothing to do.")
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	209 return ppmngr
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	210
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	211 def _loadSources(self, ppmngr):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	212 for ppinfo in ppmngr.getPipelineInfos():
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	213 rec = ppinfo.record_history.current
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	214 rec_entries = ppinfo.pipeline.loadAllContents()
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	215 if rec_entries is not None:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	216 for e in rec_entries:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	217 rec.addEntry(e)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	218
1014 071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	219 def _startPopulateTemplateCaches(self, pool):
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	220 # If we can, cache templates in a worker process, so we can load
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	221 # the sources' pages in the main process in the meantime.
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	222 # But if we don't have any workers, well, we'll have to make do
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	223 # in the `_endPopulateTemplateCache` method.
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	224 if pool.pool_size == 0:
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	225 return
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1007 diff changeset	226
1014 071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	227 pool._callback = None
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	228 pool._error_callback = None
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	229 job = {'job_spec': ('__special__', 'populate_template_cache')}
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	230 pool.queueJobs([job])
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1007 diff changeset	231
1014 071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	232 def _endPopulateTemplateCache(self, pool):
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	233 if pool.pool_size == 0:
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	234 # No workers... load the templates synchronously.
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	235 for eng in self.app.plugin_loader.getTemplateEngines():
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	236 eng.populateCache()
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	237 else:
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	238 # Wait for the job to finish.
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	239 pool.wait()
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	240 pool._callback = self._handleWorkerResult
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	241 pool._error_callback = self._handleWorkerError
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1007 diff changeset	242
972 bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	243 def _bakeRealms(self, pool, ppmngr, record_histories):
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	244 # Bake the realms -- user first, theme second, so that a user item
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	245 # can override a theme item.
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	246 # Do this for as many times as we have pipeline passes left to do.
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	247 realm_list = [REALM_USER, REALM_THEME]
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	248 pp_by_pass_and_realm = _get_pipeline_infos_by_pass_and_realm(
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	249 ppmngr.getPipelineInfos())
972 bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	250
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	251 for pp_pass_num in sorted(pp_by_pass_and_realm.keys()):
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	252 logger.debug("Pipelines pass %d" % pp_pass_num)
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	253 pp_by_realm = pp_by_pass_and_realm[pp_pass_num]
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	254 for realm in realm_list:
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	255 pplist = pp_by_realm.get(realm)
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	256 if pplist is not None:
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	257 self._bakeRealm(pool, ppmngr, record_histories,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	258 pp_pass_num, realm, pplist)
972 bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	259
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	260 def _bakeRealm(self, pool, ppmngr, record_histories,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	261 pp_pass_num, realm, pplist):
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	262 # Start with the first step, where we iterate on the content sources'
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	263 # items and run jobs on those.
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	264 pool.userdata.cur_step = 0
d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	265 next_step_jobs = {}
d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	266 pool.userdata.next_step_jobs = next_step_jobs
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	267
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	268 start_time = time.perf_counter()
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	269 job_count = 0
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	270 stats = self.app.env.stats
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	271 realm_name = REALM_NAMES[realm].lower()
1007 09dc0240f08a bake: Simplify output. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	272 participating_source_names = []
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	273
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	274 for ppinfo in pplist:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	275 src = ppinfo.source
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	276 pp = ppinfo.pipeline
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	277 jcctx = PipelineJobCreateContext(pp_pass_num, pp.record_name,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	278 record_histories)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	279
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	280 next_step_jobs[src.name] = []
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	281 jobs = pp.createJobs(jcctx)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	282 if jobs is not None:
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 972 diff changeset	283 new_job_count = len(jobs)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 972 diff changeset	284 job_count += new_job_count
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	285 pool.queueJobs(jobs)
1007 09dc0240f08a bake: Simplify output. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	286 participating_source_names.append(src.name)
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 972 diff changeset	287 else:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 972 diff changeset	288 new_job_count = 0
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 972 diff changeset	289
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 972 diff changeset	290 logger.debug(
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 972 diff changeset	291 "Queued %d jobs for source '%s' using pipeline '%s' "
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 972 diff changeset	292 "(%s, step 0)." %
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 972 diff changeset	293 (new_job_count, src.name, pp.PIPELINE_NAME, realm_name))
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	294
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	295 if job_count == 0:
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	296 logger.debug("No jobs queued! Bailing out of this bake pass.")
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	297 return
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	298
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	299 pool.wait()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	300
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	301 logger.info(format_timed(
1007 09dc0240f08a bake: Simplify output. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	302 start_time, "%d jobs completed (%s)." %
09dc0240f08a bake: Simplify output. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	303 (job_count, ', '.join(participating_source_names))))
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	304
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	305 # Now let's see if any job created a follow-up job. Let's keep
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	306 # processing those jobs as long as they create new ones.
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	307 pool.userdata.cur_step = 1
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	308 while True:
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	309 # Make a copy of out next step jobs and reset the list, so
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	310 # the first jobs to be processed don't mess it up as we're
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	311 # still iterating on it.
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	312 next_step_jobs = pool.userdata.next_step_jobs
d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	313 pool.userdata.next_step_jobs = {}
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	314
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	315 start_time = time.perf_counter()
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	316 job_count = 0
1007 09dc0240f08a bake: Simplify output. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	317 participating_source_names = []
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	318
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	319 for sn, jobs in next_step_jobs.items():
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	320 if jobs:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	321 logger.debug(
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	322 "Queuing jobs for source '%s' (%s, step %d)." %
d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	323 (sn, realm_name, pool.userdata.cur_step))
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	324
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	325 pp = ppmngr.getPipeline(sn)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	326 valctx = PipelineJobValidateContext(
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	327 pp_pass_num, pool.userdata.cur_step,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	328 pp.record_name, record_histories)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	329 pp.validateNextStepJobs(jobs, valctx)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	330
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	331 job_count += len(jobs)
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	332 pool.userdata.next_step_jobs[sn] = []
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	333 pool.queueJobs(jobs)
1007 09dc0240f08a bake: Simplify output. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	334 participating_source_names.append(sn)
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	335
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	336 if job_count == 0:
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	337 break
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	338
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	339 pool.wait()
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	340
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	341 logger.info(format_timed(
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	342 start_time,
1007 09dc0240f08a bake: Simplify output. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	343 "%d jobs completed (%s)." %
09dc0240f08a bake: Simplify output. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	344 (job_count, ', '.join(participating_source_names))))
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 868 diff changeset	345
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	346 pool.userdata.cur_step += 1
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	347
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	348 def _logErrors(self, item_spec, errors):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	349 logger.error("Errors found in %s:" % item_spec)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	350 for e in errors:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	351 logger.error(" " + e)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	352
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	353 def _logWorkerException(self, item_spec, exc_data):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	354 logger.error("Errors found in %s:" % item_spec)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	355 logger.error(exc_data['value'])
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	356 if self.app.debug:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	357 logger.error(exc_data['traceback'])
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	358
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	359 def _createWorkerPool(self, previous_records_path, pool_userdata):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	360 from piecrust.workerpool import WorkerPool
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	361 from piecrust.baking.worker import BakeWorkerContext, BakeWorker
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	362
456 5e902e228053 bake: Correctly use the `num_worers` setting. Ludovic Chabant <ludovic@chabant.com> parents: 453 diff changeset	363 worker_count = self.app.config.get('baker/workers')
462 04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 456 diff changeset	364 batch_size = self.app.config.get('baker/batch_size')
456 5e902e228053 bake: Correctly use the `num_worers` setting. Ludovic Chabant <ludovic@chabant.com> parents: 453 diff changeset	365
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	366 ctx = BakeWorkerContext(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	367 self.appfactory,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	368 self.out_dir,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	369 force=self.force,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	370 previous_records_path=previous_records_path,
868 8d25f76fce98 bake: Add ability to specify pipelines to exclude during the bake. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	371 allowed_pipelines=self.allowed_pipelines,
8d25f76fce98 bake: Add ability to specify pipelines to exclude during the bake. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	372 forbidden_pipelines=self.forbidden_pipelines)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	373 pool = WorkerPool(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	374 worker_count=worker_count,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	375 batch_size=batch_size,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	376 worker_class=BakeWorker,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	377 initargs=(ctx,),
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	378 callback=self._handleWorkerResult,
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	379 error_callback=self._handleWorkerError,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	380 userdata=pool_userdata)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	381 return pool
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	382
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	383 def _handleWorkerResult(self, job, res, userdata):
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	384 cur_step = userdata.cur_step
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	385 source_name, item_spec = job['job_spec']
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	386
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	387 # See if there's a next step to take.
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	388 npj = res.get('next_step_job')
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	389 if npj is not None:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	390 npj['step_num'] = cur_step + 1
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	391 userdata.next_step_jobs[source_name].append(npj)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	392
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	393 # Make the pipeline do custom handling to update the record entry.
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	394 ppinfo = userdata.ppmngr.getPipelineInfo(source_name)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	395 pipeline = ppinfo.pipeline
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	396 record = ppinfo.current_record
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	397 ppmrctx = PipelineJobResultHandleContext(record, job, cur_step)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	398 pipeline.handleJobResult(res, ppmrctx)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	399
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	400 # Set the overall success flags if there was an error.
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	401 record_entry = ppmrctx.record_entry
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	402 if not record_entry.success:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	403 record.success = False
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	404 userdata.records.success = False
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	405 self._logErrors(job['item_spec'], record_entry.errors)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	406
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	407 def _handleWorkerError(self, job, exc_data, userdata):
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	408 # Set the overall success flag.
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	409 source_name, item_spec = job['job_spec']
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	410 ppinfo = userdata.ppmngr.getPipelineInfo(source_name)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	411 pipeline = ppinfo.pipeline
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	412 record = ppinfo.current_record
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	413 record.success = False
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	414 userdata.records.success = False
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	415
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	416 # Add those errors to the record, if possible.
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	417 record_entry_spec = job.get('record_entry_spec', item_spec)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	418 e = record.getEntry(record_entry_spec)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	419 if e:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	420 e.errors.append(exc_data['value'])
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	421 self._logWorkerException(item_spec, exc_data)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	422
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	423 # Log debug stuff.
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	424 if self.app.debug:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	425 logger.error(exc_data.traceback)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 753 diff changeset	426
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	427
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	428 class _PoolUserData:
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	429 def __init__(self, baker, ppmngr):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	430 self.baker = baker
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	431 self.ppmngr = ppmngr
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	432 self.records = ppmngr.record_histories.current
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	433 self.cur_step = 0
d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	434 self.next_step_jobs = {}
876 d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	435
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	436
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	437 def _get_pipeline_infos_by_pass_and_realm(pp_infos):
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	438 pp_by_pass_and_realm = {}
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	439 for pp_info in pp_infos:
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	440 pp_pass_num = pp_info.pipeline.PASS_NUM
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	441 if isinstance(pp_pass_num, list):
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	442 for ppn in pp_pass_num:
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	443 _add_pipeline_info_to_pass_and_realm_dict(
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	444 ppn, pp_info, pp_by_pass_and_realm)
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	445 else:
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	446 _add_pipeline_info_to_pass_and_realm_dict(
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	447 pp_pass_num, pp_info, pp_by_pass_and_realm)
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	448 return pp_by_pass_and_realm
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	449
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	450
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	451 def _add_pipeline_info_to_pass_and_realm_dict(pp_pass_num, pp_info,
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	452 pp_by_pass_and_realm):
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	453 pp_by_realm = pp_by_pass_and_realm.setdefault(pp_pass_num, {})
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	454 pplist = pp_by_realm.setdefault(
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	455 pp_info.pipeline.source.config['realm'], [])
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	456 pplist.append(pp_info)
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	457
972 bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	458
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	459 def _merge_execution_stats(base_stats, *other_stats):
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	460 total_stats = ExecutionStats()
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	461 total_stats.mergeStats(base_stats)
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	462 for ps in other_stats:
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	463 if ps is not None:
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	464 total_stats.mergeStats(ps)
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	465 return total_stats
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	466
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	467
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	468 def _save_bake_records(records, records_path, *, rotate_previous):
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	469 if rotate_previous:
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	470 records_dir, records_fn = os.path.split(records_path)
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	471 records_id, _ = os.path.splitext(records_fn)
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	472 for i in range(8, -1, -1):
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	473 suffix = '' if i == 0 else '.%d' % i
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	474 records_path_i = os.path.join(
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	475 records_dir,
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	476 '%s%s.records' % (records_id, suffix))
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	477 if os.path.exists(records_path_i):
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	478 records_path_next = os.path.join(
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	479 records_dir,
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	480 '%s.%s.records' % (records_id, i + 1))
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	481 if os.path.exists(records_path_next):
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	482 os.remove(records_path_next)
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	483 os.rename(records_path_i, records_path_next)
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	484
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	485 with format_timed_scope(logger, "saved bake records.",
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	486 level=logging.DEBUG, colored=False):
bbf5a96b56db internal: Clean up baker code. Ludovic Chabant <ludovic@chabant.com> parents: 918 diff changeset	487 records.save(records_path)

Mercurial > piecrust2

annotate piecrust/baking/baker.py @ 1014:071f30aa04bb