piecrust2: piecrust/baking/worker.py annotate

annotate piecrust/baking/worker.py @ 440:32c7c2d219d2

performance: Refactor how data is managed to reduce copying. * Make use of `collections.abc.Mapping` to better identify things that are supposed to look like dictionaries. * Instead of handling "overlay" of data in a dict tree in each different data object, make all objects `Mapping`s and handle merging at a higher level with the new `MergedMapping` object. * Since this new object is read-only, remove the need for deep-copying of app and page configurations. * Split data classes into separate modules.

author	Ludovic Chabant <ludovic@chabant.com>
date	Sun, 28 Jun 2015 08:22:39 -0700
parents	21e26ed867b6
children	dc8518c51cbe

rev	line source
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import time
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import copy
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import queue
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import logging
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 from piecrust.app import PieCrust
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 from piecrust.baking.single import PageBaker, BakingError
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	7 from piecrust.rendering import (
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8 QualifiedPage, PageRenderingContext, render_page_segments)
430 21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	9 from piecrust.routing import create_route_metadata
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	10 from piecrust.sources.base import PageFactory
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	11
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13 logger = logging.getLogger(__name__)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	14
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	15
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16 def worker_func(wid, ctx):
426 ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	17 if ctx.is_profiling:
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	18 try:
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	19 import cProfile as profile
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	20 except ImportError:
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	21 import profile
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	22
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	23 ctx.is_profiling = False
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	24 profile.runctx('_real_worker_func(wid, ctx)',
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	25 globals(), locals(),
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	26 filename='BakeWorker-%d.prof' % wid)
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	27 else:
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	28 _real_worker_func(wid, ctx)
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	29
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	30
ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	31 def _real_worker_func(wid, ctx):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32 logger.debug("Worker %d booting up..." % wid)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 w = BakeWorker(wid, ctx)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34 w.run()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37 class BakeWorkerContext(object):
425 afeebdd9f767 bake: Pass the sub-cache directory to the bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 422 diff changeset	38 def __init__(self, root_dir, sub_cache_dir, out_dir,
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	39 work_queue, results, abort_event,
426 ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	40 force=False, debug=False, is_profiling=False):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41 self.root_dir = root_dir
425 afeebdd9f767 bake: Pass the sub-cache directory to the bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 422 diff changeset	42 self.sub_cache_dir = sub_cache_dir
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	43 self.out_dir = out_dir
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44 self.work_queue = work_queue
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	45 self.results = results
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	46 self.abort_event = abort_event
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	47 self.force = force
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	48 self.debug = debug
426 ed5ccd4cce49 performance: Quick and dirty profiling support for bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 425 diff changeset	49 self.is_profiling = is_profiling
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52 JOB_LOAD, JOB_RENDER_FIRST, JOB_BAKE = range(0, 3)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	55 class BakeWorkerJob(object):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	56 def __init__(self, job_type, payload):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	57 self.job_type = job_type
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58 self.payload = payload
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	60
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	61 class BakeWorker(object):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62 def __init__(self, wid, ctx):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63 self.wid = wid
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64 self.ctx = ctx
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	65
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	66 def run(self):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	67 logger.debug("Working %d initializing..." % self.wid)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 work_start_time = time.perf_counter()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	70 # Create the app local to this worker.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71 app = PieCrust(self.ctx.root_dir, debug=self.ctx.debug)
425 afeebdd9f767 bake: Pass the sub-cache directory to the bake workers. Ludovic Chabant <ludovic@chabant.com> parents: 422 diff changeset	72 app._useSubCacheDir(self.ctx.sub_cache_dir)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	73 app.env.fs_cache_only_for_main_page = True
421 4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	74 app.env.registerTimer("BakeWorker_%d_Total" % self.wid)
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	75 app.env.registerTimer("BakeWorkerInit")
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76 app.env.registerTimer("JobReceive")
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	77
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	78 # Create the job handlers.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	79 job_handlers = {
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80 JOB_LOAD: LoadJobHandler(app, self.ctx),
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	81 JOB_RENDER_FIRST: RenderFirstSubJobHandler(app, self.ctx),
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	82 JOB_BAKE: BakeJobHandler(app, self.ctx)}
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	83 for jt, jh in job_handlers.items():
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84 app.env.registerTimer(type(jh).__name__)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	85
421 4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	86 app.env.stepTimerSince("BakeWorkerInit", work_start_time)
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	87
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	88 # Start working!
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	89 aborted_with_exception = None
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	90 while not self.ctx.abort_event.is_set():
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	91 try:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	92 with app.env.timerScope('JobReceive'):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	93 job = self.ctx.work_queue.get(True, 0.01)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	94 except queue.Empty:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	95 continue
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	96
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	97 try:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	98 handler = job_handlers[job.job_type]
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	99 with app.env.timerScope(type(handler).__name__):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	100 handler.handleJob(job)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	101 except Exception as ex:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	102 self.ctx.abort_event.set()
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	103 aborted_with_exception = ex
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	104 logger.debug("[%d] Critical error, aborting." % self.wid)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	105 if self.ctx.debug:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	106 logger.exception(ex)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	107 break
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	108 finally:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	109 self.ctx.work_queue.task_done()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	110
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	111 if aborted_with_exception is not None:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	112 msgs = _get_errors(aborted_with_exception)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	113 self.ctx.results.put_nowait({'type': 'error', 'messages': msgs})
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	114
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	115 # Send our timers to the main process before exiting.
421 4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	116 app.env.stepTimerSince("BakeWorker_%d_Total" % self.wid,
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	117 work_start_time)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	118 self.ctx.results.put_nowait({
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	119 'type': 'timers', 'data': app.env._timers})
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	120
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	121
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	122 class JobHandler(object):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	123 def __init__(self, app, ctx):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	124 self.app = app
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	125 self.ctx = ctx
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	126
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	127 def handleJob(self, job):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	128 raise NotImplementedError()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	129
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	130
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	131 def _get_errors(ex):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	132 errors = []
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	133 while ex is not None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	134 errors.append(str(ex))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	135 ex = ex.__cause__
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	136 return errors
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	137
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	138
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	139 class PageFactoryInfo(object):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	140 def __init__(self, fac):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	141 self.source_name = fac.source.name
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	142 self.rel_path = fac.rel_path
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	143 self.metadata = fac.metadata
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	144
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	145 def build(self, app):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	146 source = app.getSource(self.source_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	147 return PageFactory(source, self.rel_path, self.metadata)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	148
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	149
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150 class LoadJobPayload(object):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	151 def __init__(self, fac):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	152 self.factory_info = PageFactoryInfo(fac)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	153
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	154
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	155 class LoadJobResult(object):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	156 def __init__(self, source_name, path):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	157 self.source_name = source_name
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	158 self.path = path
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	159 self.config = None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	160 self.errors = None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	161
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	162
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	163 class RenderFirstSubJobPayload(object):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	164 def __init__(self, fac):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	165 self.factory_info = PageFactoryInfo(fac)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	166
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	167
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	168 class RenderFirstSubJobResult(object):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	169 def __init__(self, path):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	170 self.path = path
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	171 self.errors = None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	172
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	173
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	174 class BakeJobPayload(object):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	175 def __init__(self, fac, route_metadata, previous_entry,
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	176 dirty_source_names, tax_info=None):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	177 self.factory_info = PageFactoryInfo(fac)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	178 self.route_metadata = route_metadata
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	179 self.previous_entry = previous_entry
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	180 self.dirty_source_names = dirty_source_names
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	181 self.taxonomy_info = tax_info
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	182
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	183
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	184 class BakeJobResult(object):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	185 def __init__(self, path, tax_info=None):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	186 self.path = path
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	187 self.taxonomy_info = tax_info
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	188 self.sub_entries = None
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	189 self.errors = None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	190
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	191
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	192 class LoadJobHandler(JobHandler):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	193 def handleJob(self, job):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	194 # Just make sure the page has been cached.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	195 fac = job.payload.factory_info.build(self.app)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	196 logger.debug("Loading page: %s" % fac.ref_spec)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	197 result = LoadJobResult(fac.source.name, fac.path)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	198 try:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	199 page = fac.buildPage()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	200 page._load()
440 32c7c2d219d2 performance: Refactor how data is managed to reduce copying. Ludovic Chabant <ludovic@chabant.com> parents: 430 diff changeset	201 result.config = page.config.getAll()
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	202 except Exception as ex:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	203 logger.debug("Got loading error. Sending it to master.")
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	204 result.errors = _get_errors(ex)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	205 if self.ctx.debug:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	206 logger.exception(ex)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	207
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	208 self.ctx.results.put_nowait(result)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	209
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	210
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	211 class RenderFirstSubJobHandler(JobHandler):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	212 def handleJob(self, job):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	213 # Render the segments for the first sub-page of this page.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	214 fac = job.payload.factory_info.build(self.app)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	215
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	216 # These things should be OK as they're checked upstream by the baker.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	217 route = self.app.getRoute(fac.source.name, fac.metadata,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	218 skip_taxonomies=True)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	219 assert route is not None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	220
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	221 page = fac.buildPage()
430 21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	222 route_metadata = create_route_metadata(page)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	223 qp = QualifiedPage(page, route, route_metadata)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	224 ctx = PageRenderingContext(qp)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	225
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	226 result = RenderFirstSubJobResult(fac.path)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	227 logger.debug("Preparing page: %s" % fac.ref_spec)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	228 try:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	229 render_page_segments(ctx)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	230 except Exception as ex:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	231 logger.debug("Got rendering error. Sending it to master.")
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	232 result.errors = _get_errors(ex)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	233 if self.ctx.debug:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	234 logger.exception(ex)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	235
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	236 self.ctx.results.put_nowait(result)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	237
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	238
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	239 class BakeJobHandler(JobHandler):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	240 def __init__(self, app, ctx):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	241 super(BakeJobHandler, self).__init__(app, ctx)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	242 self.page_baker = PageBaker(app, ctx.out_dir, ctx.force)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	243
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	244 def handleJob(self, job):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	245 # Actually bake the page and all its sub-pages to the output folder.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	246 fac = job.payload.factory_info.build(self.app)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	247
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	248 route_metadata = job.payload.route_metadata
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	249 tax_info = job.payload.taxonomy_info
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	250 if tax_info is not None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	251 route = self.app.getTaxonomyRoute(tax_info.taxonomy_name,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	252 tax_info.source_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	253 else:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	254 route = self.app.getRoute(fac.source.name, route_metadata,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	255 skip_taxonomies=True)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	256 assert route is not None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	257
430 21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	258 page = fac.buildPage()
21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	259 qp = QualifiedPage(page, route, route_metadata)
21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	260
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	261 result = BakeJobResult(fac.path, tax_info)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	262 previous_entry = job.payload.previous_entry
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	263 dirty_source_names = job.payload.dirty_source_names
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	264 logger.debug("Baking page: %s" % fac.ref_spec)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	265 try:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	266 sub_entries = self.page_baker.bake(
430 21e26ed867b6 internal: Create full route metadata in one place. Ludovic Chabant <ludovic@chabant.com> parents: 426 diff changeset	267 qp, previous_entry, dirty_source_names, tax_info)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	268 result.sub_entries = sub_entries
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	269
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	270 except BakingError as ex:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	271 logger.debug("Got baking error. Sending it to master.")
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	272 result.errors = _get_errors(ex)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	273 if self.ctx.debug:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	274 logger.exception(ex)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	275
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	276 self.ctx.results.put_nowait(result)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	277

Mercurial > piecrust2

annotate piecrust/baking/worker.py @ 440:32c7c2d219d2