piecrust2: piecrust/baking/single.py annotate

annotate piecrust/baking/single.py @ 412:a1567766c83c

internal: Allow re-registering performance timers.

author	Ludovic Chabant <ludovic@chabant.com>
date	Sat, 20 Jun 2015 19:15:57 -0700
parents	e7b865f8f335
children	0e9a94b7fdfa

rev	line source
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import os.path
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import shutil
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import codecs
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import logging
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import urllib.parse
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 329 diff changeset	6 from piecrust.baking.records import (
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	7 PageBakeInfo, SubPageBakeInfo, BakePassInfo)
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	8 from piecrust.rendering import (
369 4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 350 diff changeset	9 QualifiedPage, PageRenderingContext, render_page,
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	10 PASS_FORMATTING)
329 422052d2e978 internal: Try handling URLs in a consistent way. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	11 from piecrust.uriutil import split_uri
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	14 logger = logging.getLogger(__name__)
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	15
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16
264 74bea91c9630 bake: Don't store internal config values in the bake record. Ludovic Chabant <ludovic@chabant.com> parents: 262 diff changeset	17 def copy_public_page_config(config):
369 4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 350 diff changeset	18 res = config.getDeepcopy()
264 74bea91c9630 bake: Don't store internal config values in the bake record. Ludovic Chabant <ludovic@chabant.com> parents: 262 diff changeset	19 for k in list(res.keys()):
74bea91c9630 bake: Don't store internal config values in the bake record. Ludovic Chabant <ludovic@chabant.com> parents: 262 diff changeset	20 if k.startswith('__'):
74bea91c9630 bake: Don't store internal config values in the bake record. Ludovic Chabant <ludovic@chabant.com> parents: 262 diff changeset	21 del res[k]
74bea91c9630 bake: Don't store internal config values in the bake record. Ludovic Chabant <ludovic@chabant.com> parents: 262 diff changeset	22 return res
74bea91c9630 bake: Don't store internal config values in the bake record. Ludovic Chabant <ludovic@chabant.com> parents: 262 diff changeset	23
74bea91c9630 bake: Don't store internal config values in the bake record. Ludovic Chabant <ludovic@chabant.com> parents: 262 diff changeset	24
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	25 class BakingError(Exception):
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26 pass
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29 class PageBaker(object):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	30 def __init__(self, app, out_dir, force=False, copy_assets=True):
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31 self.app = app
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32 self.out_dir = out_dir
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 self.force = force
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34 self.copy_assets = copy_assets
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35 self.site_root = app.config.get('site/root')
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 self.pretty_urls = app.config.get('site/pretty_urls')
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	38 def getOutputPath(self, uri):
329 422052d2e978 internal: Try handling URLs in a consistent way. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	39 uri_root, uri_path = split_uri(self.app, uri)
422052d2e978 internal: Try handling URLs in a consistent way. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	40
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41 bake_path = [self.out_dir]
329 422052d2e978 internal: Try handling URLs in a consistent way. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	42 decoded_uri = urllib.parse.unquote(uri_path)
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	43 if self.pretty_urls:
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44 bake_path.append(decoded_uri)
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	45 bake_path.append('index.html')
262 61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 256 diff changeset	46 elif decoded_uri == '':
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 256 diff changeset	47 bake_path.append('index.html')
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	48 else:
262 61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 256 diff changeset	49 bake_path.append(decoded_uri)
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51 return os.path.normpath(os.path.join(*bake_path))
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	53 def bake(self, factory, route, route_metadata, prev_entry,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	54 first_render_info, dirty_source_names, tax_info=None):
369 4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 350 diff changeset	55 # Get the page.
4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 350 diff changeset	56 page = factory.buildPage()
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	57
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	58 # Start baking the sub-pages.
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 cur_sub = 1
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	60 has_more_subs = True
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	61 report = PageBakeInfo()
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63 while has_more_subs:
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	64 # Get the URL and path for this sub-page.
256 da5e6e00fb41 bake/serve: Make previewed and baked URLs consistent. Ludovic Chabant <ludovic@chabant.com> parents: 235 diff changeset	65 sub_uri = route.getUri(route_metadata, sub_num=cur_sub,
329 422052d2e978 internal: Try handling URLs in a consistent way. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	66 provider=page)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	67 logger.debug("Baking '%s' [%d]..." % (sub_uri, cur_sub))
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 out_path = self.getOutputPath(sub_uri)
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	70 # Create the sub-entry for the bake record.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	71 sub_entry = SubPageBakeInfo(sub_uri, out_path)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	72 report.subs.append(sub_entry)
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	73
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	74 # Find a corresponding sub-entry in the previous bake record.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	75 prev_sub_entry = None
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	76 if prev_entry:
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	77 try:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	78 prev_sub_entry = prev_entry.getSub(cur_sub)
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	79 except IndexError:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	80 pass
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	81
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	82 # Figure out if we need to invalidate or force anything.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	83 force_this_sub, invalidate_formatting = _compute_force_flags(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	84 prev_sub_entry, sub_entry, dirty_source_names)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	85 force_this_sub = force_this_sub or self.force
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	86
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	87 # Check for up-to-date outputs.
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	88 do_bake = True
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	89 if not force_this_sub:
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	90 try:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 329 diff changeset	91 in_path_time = page.path_mtime
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	92 out_path_time = os.path.getmtime(out_path)
286 a2d283d1033d tests: Fixes for running on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 280 diff changeset	93 if out_path_time >= in_path_time:
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	94 do_bake = False
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	95 except OSError:
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	96 # File doesn't exist, we'll need to bake.
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	97 pass
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	98
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	99 # If this page didn't bake because it's already up-to-date.
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	100 # Keep trying for as many subs as we know this page has.
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	101 if not do_bake:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	102 prev_sub_entry.collapseRenderPasses(sub_entry)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	103 sub_entry.flags = SubPageBakeInfo.FLAG_NONE
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	104
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	105 if prev_entry.num_subs >= cur_sub + 1:
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	106 cur_sub += 1
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	107 has_more_subs = True
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	108 logger.debug(" %s is up to date, skipping to next "
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 329 diff changeset	109 "sub-page." % out_path)
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	110 continue
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	111
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	112 logger.debug(" %s is up to date, skipping bake." % out_path)
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	113 break
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	114
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	115 # All good, proceed.
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	116 try:
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	117 if invalidate_formatting:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 329 diff changeset	118 cache_key = sub_uri
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	119 self.app.env.rendered_segments_repository.invalidate(
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	120 cache_key)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	121 sub_entry.flags \|= \
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	122 SubPageBakeInfo.FLAG_FORMATTING_INVALIDATED
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	123
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	124 logger.debug(" p%d -> %s" % (cur_sub, out_path))
369 4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 350 diff changeset	125 qp = QualifiedPage(page, route, route_metadata)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	126 ctx, rp = self._bakeSingle(qp, cur_sub, out_path, tax_info)
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	127 except Exception as ex:
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	128 if self.app.debug:
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	129 logger.exception(ex)
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	130 page_rel_path = os.path.relpath(page.path, self.app.root_dir)
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	131 raise BakingError("%s: error baking '%s'." %
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	132 (page_rel_path, sub_uri)) from ex
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	133
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	134 # Record what we did.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	135 sub_entry.flags \|= SubPageBakeInfo.FLAG_BAKED
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	136 # self.record.dirty_source_names.add(record_entry.source_name)
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	137 for p, pinfo in ctx.render_passes.items():
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	138 bpi = BakePassInfo()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	139 bpi.used_source_names = set(pinfo.used_source_names)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	140 bpi.used_taxonomy_terms = set(pinfo.used_taxonomy_terms)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	141 sub_entry.render_passes[p] = bpi
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	142 if prev_sub_entry:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	143 prev_sub_entry.collapseRenderPasses(sub_entry)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	144
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	145 # If this page has had its first sub-page rendered already, we
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	146 # have that information from the baker. Otherwise (e.g. for
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	147 # taxonomy pages), we have that information from the result
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	148 # of the render.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	149 info = ctx
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	150 if cur_sub == 1 and first_render_info is not None:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	151 info = first_render_info
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	152
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	153 # Copy page assets.
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	154 if cur_sub == 1 and self.copy_assets and info.used_assets:
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	155 if self.pretty_urls:
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	156 out_assets_dir = os.path.dirname(out_path)
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	157 else:
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	158 out_assets_dir, out_name = os.path.split(out_path)
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	159 if sub_uri != self.site_root:
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	160 out_name_noext, _ = os.path.splitext(out_name)
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	161 out_assets_dir += out_name_noext
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	162
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	163 logger.debug("Copying page assets to: %s" % out_assets_dir)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	164 _ensure_dir_exists(out_assets_dir)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	165
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	166 used_assets = info.used_assets
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	167 for ap in used_assets:
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	168 dest_ap = os.path.join(out_assets_dir,
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	169 os.path.basename(ap))
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	170 logger.debug(" %s -> %s" % (ap, dest_ap))
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	171 shutil.copy(ap, dest_ap)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	172 report.assets.append(ap)
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	173
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	174 # Figure out if we have more work.
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	175 has_more_subs = False
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	176 if info.pagination_has_more:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	177 cur_sub += 1
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	178 has_more_subs = True
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	179
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	180 return report
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	181
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	182 def _bakeSingle(self, qualified_page, num, out_path, tax_info=None):
369 4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 350 diff changeset	183 ctx = PageRenderingContext(qualified_page, page_num=num)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	184 if tax_info:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	185 tax = self.app.getTaxonomy(tax_info.taxonomy_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	186 ctx.setTaxonomyFilter(tax, tax_info.term)
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	187
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	188 rp = render_page(ctx)
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	189
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	190 out_dir = os.path.dirname(out_path)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	191 _ensure_dir_exists(out_dir)
150 91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	192
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	193 with codecs.open(out_path, 'w', 'utf8') as fp:
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	194 fp.write(rp.content)
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	195
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	196 return ctx, rp
91dcbb5fe1e8 Split baking code in smaller files. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	197
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	198
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	199 def _compute_force_flags(prev_sub_entry, sub_entry, dirty_source_names):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	200 # Figure out what to do with this page.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	201 force_this_sub = False
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	202 invalidate_formatting = False
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	203 sub_uri = sub_entry.out_uri
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	204 if (prev_sub_entry and
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	205 (prev_sub_entry.was_baked_successfully or
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	206 prev_sub_entry.was_clean)):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	207 # If the current page is known to use pages from other sources,
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	208 # see if any of those got baked, or are going to be baked for
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	209 # some reason. If so, we need to bake this one too.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	210 # (this happens for instance with the main page of a blog).
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	211 dirty_for_this, invalidated_render_passes = (
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	212 _get_dirty_source_names_and_render_passes(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	213 prev_sub_entry, dirty_source_names))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	214 if len(invalidated_render_passes) > 0:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	215 logger.debug(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	216 "'%s' is known to use sources %s, which have "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	217 "items that got (re)baked. Will force bake this "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	218 "page. " % (sub_uri, dirty_for_this))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	219 sub_entry.flags \|= \
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	220 SubPageBakeInfo.FLAG_FORCED_BY_SOURCE
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	221 force_this_sub = True
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	222
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	223 if PASS_FORMATTING in invalidated_render_passes:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	224 logger.debug(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	225 "Will invalidate cached formatting for '%s' "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	226 "since sources were using during that pass."
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	227 % sub_uri)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	228 invalidate_formatting = True
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	229 elif (prev_sub_entry and
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	230 prev_sub_entry.errors):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	231 # Previous bake failed. We'll have to bake it again.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	232 logger.debug(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	233 "Previous record entry indicates baking failed for "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	234 "'%s'. Will bake it again." % sub_uri)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	235 sub_entry.flags \|= \
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	236 SubPageBakeInfo.FLAG_FORCED_BY_PREVIOUS_ERRORS
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	237 force_this_sub = True
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	238 elif not prev_sub_entry:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	239 # No previous record. We'll have to bake it.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	240 logger.debug("No previous record entry found for '%s'. Will "
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	241 "force bake it." % sub_uri)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	242 sub_entry.flags \|= \
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	243 SubPageBakeInfo.FLAG_FORCED_BY_NO_PREVIOUS
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	244 force_this_sub = True
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	245
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	246 return force_this_sub, invalidate_formatting
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	247
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	248
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	249 def _get_dirty_source_names_and_render_passes(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	250 sub_entry, dirty_source_names):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	251 dirty_for_this = set()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	252 invalidated_render_passes = set()
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	253 for p, pinfo in sub_entry.render_passes.items():
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	254 for src_name in pinfo.used_source_names:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	255 is_dirty = (src_name in dirty_source_names)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	256 if is_dirty:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	257 invalidated_render_passes.add(p)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	258 dirty_for_this.add(src_name)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	259 break
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	260 return dirty_for_this, invalidated_render_passes
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	261
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	262
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	263 def _ensure_dir_exists(path):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	264 try:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	265 os.makedirs(path, mode=0o755, exist_ok=True)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	266 except OSError:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	267 # In a multiprocess environment, several process may very
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	268 # occasionally try to create the same directory at the same time.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	269 # Let's ignore any error and if something's really wrong (like file
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	270 # acces permissions or whatever), then it will more legitimately fail
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	271 # just after this when we try to write files.
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	272 pass
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 369 diff changeset	273

Mercurial > piecrust2

annotate piecrust/baking/single.py @ 412:a1567766c83c