piecrust2: piecrust/commands/builtin/baking.py annotate

annotate piecrust/commands/builtin/baking.py @ 666:81d9c3a3a0b5

internal: Get rid of the whole "sub cache" business. * Compute cache keys up front, so the cache directory is only chosen once. * Buffer up config variants to apply before loading the config. Makes it possible to cache variant-resulting configs, too. * Make a factory class to reuse the logic that creates the `PieCrust` object correctly for multi-process workers and such. * Add a test.

author	Ludovic Chabant <ludovic@chabant.com>
date	Thu, 03 Mar 2016 08:22:41 -0800
parents	466bbddd121e
children	61d606fbc313

rev	line source
121 300eb1c2cb14 Fix stupid bug. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	1 import time
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os.path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import logging
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import hashlib
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	5 import fnmatch
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	6 import datetime
421 4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	7 from colorama import Fore
580 c7ac7dc2fe73 bake: Add option to bake assets for FoodTruck. This is likely temporary. Ludovic Chabant <ludovic@chabant.com> parents: 466 diff changeset	8 from piecrust import CACHE_DIR
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	9 from piecrust.baking.baker import Baker
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	10 from piecrust.baking.records import (
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	11 BakeRecord, BakeRecordEntry, SubPageBakeInfo)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	12 from piecrust.chefutil import format_timed
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13 from piecrust.commands.base import ChefCommand
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	14 from piecrust.processing.pipeline import ProcessorPipeline
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	15 from piecrust.processing.records import (
221 f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	16 ProcessorPipelineRecord,
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	17 FLAG_PREPARED, FLAG_PROCESSED,
221 f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	18 FLAG_BYPASSED_STRUCTURED_PROCESSING)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	19 from piecrust.rendering import (
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	20 PASS_FORMATTING, PASS_RENDERING)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	23 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	25
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26 class BakeCommand(ChefCommand):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 super(BakeCommand, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29 self.name = 'bake'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	30 self.description = "Bakes your website into static HTML files."
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32 def setupParser(self, parser, app):
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	33 parser.add_argument(
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	34 '-o', '--output',
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35 help="The directory to put all the baked HTML files into "
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 "(defaults to `_counter`)")
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	37 parser.add_argument(
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	38 '-f', '--force',
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	39 help="Force re-baking the entire website.",
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	40 action='store_true')
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	41 parser.add_argument(
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	42 '-w', '--workers',
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	43 help="The number of worker processes to spawn.",
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	44 type=int, default=-1)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	45 parser.add_argument(
462 04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 421 diff changeset	46 '--batch-size',
04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 421 diff changeset	47 help="The number of jobs per batch.",
04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 421 diff changeset	48 type=int, default=-1)
04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 421 diff changeset	49 parser.add_argument(
220 84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	50 '--assets-only',
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	51 help="Only bake the assets (don't bake the web pages).",
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	52 action='store_true')
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	53 parser.add_argument(
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	54 '--html-only',
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	55 help="Only bake the pages (don't run the asset pipeline).",
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	56 action='store_true')
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	57 parser.add_argument(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	58 '--show-timers',
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	59 help="Show detailed timing information.",
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	60 action='store_true')
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	61
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62 def run(self, ctx):
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	63 out_dir = (ctx.args.output or
bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	64 os.path.join(ctx.app.root_dir, '_counter'))
bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	65
215 a47580a0955b bake: Better error handling for the processing pipeline. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	66 success = True
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	67 ctx.timers = {}
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	68 start_time = time.perf_counter()
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	69 try:
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	70 # Bake the site sources.
220 84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	71 if not ctx.args.assets_only:
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	72 success = success & self._bakeSources(ctx, out_dir)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	73
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	74 # Bake the assets.
220 84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	75 if not ctx.args.html_only:
215 a47580a0955b bake: Better error handling for the processing pipeline. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	76 success = success & self._bakeAssets(ctx, out_dir)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	77
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	78 # Show merged timers.
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	79 if ctx.args.show_timers:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	80 logger.info("-------------------")
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	81 logger.info("Timing information:")
421 4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	82 _show_timers(ctx.timers)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	83
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	84 # All done.
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	85 logger.info('-------------------------')
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	86 logger.info(format_timed(start_time, 'done baking'))
215 a47580a0955b bake: Better error handling for the processing pipeline. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	87 return 0 if success else 1
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	88 except Exception as ex:
43 f14889d6b067 Proper debug logging. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	89 if ctx.app.debug:
f14889d6b067 Proper debug logging. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	90 logger.exception(ex)
f14889d6b067 Proper debug logging. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	91 else:
f14889d6b067 Proper debug logging. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	92 logger.error(str(ex))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	93 return 1
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	94
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	95 def _bakeSources(self, ctx, out_dir):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	96 if ctx.args.workers > 0:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	97 ctx.app.config.set('baker/workers', ctx.args.workers)
462 04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 421 diff changeset	98 if ctx.args.batch_size > 0:
04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 421 diff changeset	99 ctx.app.config.set('baker/batch_size', ctx.args.batch_size)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	100 baker = Baker(
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	101 ctx.app, out_dir,
466 456db44dcc53 bake: Pass the config variants and values from the CLI to the baker. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	102 force=ctx.args.force,
456db44dcc53 bake: Pass the config variants and values from the CLI to the baker. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	103 applied_config_variant=ctx.config_variant,
456db44dcc53 bake: Pass the config variants and values from the CLI to the baker. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	104 applied_config_values=ctx.config_values)
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 215 diff changeset	105 record = baker.bake()
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	106 _merge_timers(record.timers, ctx.timers)
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 215 diff changeset	107 return record.success
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	108
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	109 def _bakeAssets(self, ctx, out_dir):
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	110 proc = ProcessorPipeline(
205 e725af1d48fb bake: Changes in how assets directories are configured. Ludovic Chabant <ludovic@chabant.com> parents: 201 diff changeset	111 ctx.app, out_dir,
666 81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 653 diff changeset	112 force=ctx.args.force,
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 653 diff changeset	113 applied_config_variant=ctx.config_variant,
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 653 diff changeset	114 applied_config_values=ctx.config_values)
215 a47580a0955b bake: Better error handling for the processing pipeline. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	115 record = proc.run()
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	116 _merge_timers(record.timers, ctx.timers)
215 a47580a0955b bake: Better error handling for the processing pipeline. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	117 return record.success
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	118
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	119
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	120 def _merge_timers(source, target):
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	121 if source is None:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	122 return
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	123
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	124 for name, val in source.items():
421 4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	125 if isinstance(val, float):
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	126 if name not in target:
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	127 target[name] = 0
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	128 target[name] += val
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	129 elif isinstance(val, dict):
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	130 if name not in target:
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	131 target[name] = {}
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	132 _merge_timers(val, target[name])
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	133
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	134
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	135 def _show_timers(timers, indent=''):
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	136 sub_timer_names = []
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	137 for name in sorted(timers.keys()):
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	138 if isinstance(timers[name], float):
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	139 val_str = '%8.1f s' % timers[name]
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	140 logger.info(
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	141 "%s[%s%s%s] %s" %
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	142 (indent, Fore.GREEN, val_str, Fore.RESET, name))
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	143 else:
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	144 sub_timer_names.append(name)
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	145
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	146 for name in sub_timer_names:
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	147 logger.info('%s:' % name)
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	148 _show_timers(timers[name], indent + ' ')
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	149
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	150
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	151 class ShowRecordCommand(ChefCommand):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	152 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	153 super(ShowRecordCommand, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	154 self.name = 'showrecord'
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	155 self.description = ("Shows the bake record for a given output "
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	156 "directory.")
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	157
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	158 def setupParser(self, parser, app):
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	159 parser.add_argument(
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	160 '-o', '--output',
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	161 help="The output directory for which to show the bake record "
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	162 "(defaults to `_counter`)",
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	163 nargs='?')
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	164 parser.add_argument(
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	165 '-p', '--path',
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	166 help="A pattern that will be used to filter the relative path "
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	167 "of entries to show.")
331 f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	168 parser.add_argument(
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	169 '-t', '--out',
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	170 help="A pattern that will be used to filter the output path "
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	171 "of entries to show.")
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	172 parser.add_argument(
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	173 '--last',
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	174 type=int,
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	175 default=0,
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	176 help="Show the last Nth bake record.")
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	177 parser.add_argument(
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	178 '--html-only',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	179 action='store_true',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	180 help="Only show records for pages (not from the asset "
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	181 "pipeline).")
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	182 parser.add_argument(
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	183 '--assets-only',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	184 action='store_true',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	185 help="Only show records for assets (not from pages).")
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	186
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	187 def run(self, ctx):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	188 out_dir = ctx.args.output or os.path.join(ctx.app.root_dir, '_counter')
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	189 record_id = hashlib.md5(out_dir.encode('utf8')).hexdigest()
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	190 suffix = '' if ctx.args.last == 0 else '.%d' % ctx.args.last
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	191 record_name = '%s%s.record' % (record_id, suffix)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	192
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	193 pattern = None
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	194 if ctx.args.path:
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	195 pattern = '%s' % ctx.args.path.strip('*')
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	196
331 f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	197 out_pattern = None
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	198 if ctx.args.out:
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	199 out_pattern = '%s' % ctx.args.out.strip('*')
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	200
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	201 if not ctx.args.assets_only:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	202 self._showBakeRecord(ctx, record_name, pattern, out_pattern)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	203 if not ctx.args.html_only:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	204 self._showProcessingRecord(ctx, record_name, pattern, out_pattern)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	205
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	206 def _showBakeRecord(self, ctx, record_name, pattern, out_pattern):
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	207 # Show the bake record.
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	208 record_cache = ctx.app.cache.getCache('baker')
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	209 if not record_cache.has(record_name):
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	210 raise Exception("No record has been created for this output path. "
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	211 "Did you bake there yet?")
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	212
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	213 record = BakeRecord.load(record_cache.getCachePath(record_name))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	214 logging.info("Bake record for: %s" % record.out_dir)
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	215 logging.info("From: %s" % record_name)
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	216 logging.info("Last baked: %s" %
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	217 datetime.datetime.fromtimestamp(record.bake_time))
218 10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	218 if record.success:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	219 logging.info("Status: success")
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	220 else:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	221 logging.error("Status: failed")
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	222 logging.info("Entries:")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	223 for entry in record.entries:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	224 if pattern and not fnmatch.fnmatch(entry.path, pattern):
331 f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	225 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	226 if out_pattern and not (
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	227 any([o for o in entry.out_paths
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	228 if fnmatch.fnmatch(o, out_pattern)])):
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	229 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	230
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	231 flags = _get_flag_descriptions(
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	232 entry.flags,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	233 {
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	234 BakeRecordEntry.FLAG_NEW: 'new',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	235 BakeRecordEntry.FLAG_SOURCE_MODIFIED: 'modified',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	236 BakeRecordEntry.FLAG_OVERRIDEN: 'overriden'})
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	237
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	238 logging.info(" - ")
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	239
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	240 rel_path = os.path.relpath(entry.path, ctx.app.root_dir)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	241 logging.info(" path: %s" % rel_path)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	242 logging.info(" source: %s" % entry.source_name)
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	243 if entry.taxonomy_info:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	244 ti = entry.taxonomy_info
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	245 logging.info(" taxonomy: %s = %s (in %s)" %
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	246 (ti.taxonomy_name, ti.term, ti.source_name))
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	247 else:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	248 logging.info(" taxonomy: <none>")
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	249 logging.info(" flags: %s" % _join(flags))
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	250 logging.info(" config: %s" % entry.config)
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	251
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	252 if entry.errors:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	253 logging.error(" errors: %s" % entry.errors)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	254
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	255 logging.info(" %d sub-pages:" % len(entry.subs))
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	256 for sub in entry.subs:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	257 sub_flags = _get_flag_descriptions(
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	258 sub.flags,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	259 {
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	260 SubPageBakeInfo.FLAG_BAKED: 'baked',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	261 SubPageBakeInfo.FLAG_FORCED_BY_SOURCE:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	262 'forced by source',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	263 SubPageBakeInfo.FLAG_FORCED_BY_NO_PREVIOUS:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	264 'forced by missing previous record entry',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	265 SubPageBakeInfo.FLAG_FORCED_BY_PREVIOUS_ERRORS:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	266 'forced by previous errors',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	267 SubPageBakeInfo.FLAG_FORMATTING_INVALIDATED:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	268 'formatting invalidated'})
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	269
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	270 logging.info(" - ")
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	271 logging.info(" URL: %s" % sub.out_uri)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	272 logging.info(" path: %s" % os.path.relpath(
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	273 sub.out_path, record.out_dir))
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	274 logging.info(" flags: %s" % _join(sub_flags))
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	275
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	276 if sub.render_info:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	277 pass_names = {
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	278 PASS_FORMATTING: 'formatting pass',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	279 PASS_RENDERING: 'rendering pass'}
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	280 for p, ri in sub.render_info.items():
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	281 logging.info(" - %s" % pass_names[p])
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	282 logging.info(" used sources: %s" %
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	283 _join(ri.used_source_names))
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	284 pgn_info = 'no'
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	285 if ri.used_pagination:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	286 pgn_info = 'yes'
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	287 if ri.pagination_has_more:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	288 pgn_info += ', has more'
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	289 logging.info(" used pagination: %s", pgn_info)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	290 logging.info(" used assets: %s",
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	291 'yes' if ri.used_assets else 'no')
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	292 logging.info(" used terms: %s" %
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	293 _join(
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	294 ['%s=%s (%s)' % (tn, t, sn)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	295 for sn, tn, t in
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	296 ri.used_taxonomy_terms]))
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	297 else:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	298 logging.info(" no render info")
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	299
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	300 if sub.errors:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	301 logging.error(" errors: %s" % sub.errors)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	302
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	303 def _showProcessingRecord(self, ctx, record_name, pattern, out_pattern):
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	304 record_cache = ctx.app.cache.getCache('proc')
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	305 if not record_cache.has(record_name):
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	306 raise Exception("No record has been created for this output path. "
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	307 "Did you bake there yet?")
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	308
218 10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	309 # Show the pipeline record.
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	310 record = ProcessorPipelineRecord.load(
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	311 record_cache.getCachePath(record_name))
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	312 logging.info("")
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	313 logging.info("Processing record for: %s" % record.out_dir)
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	314 logging.info("Last baked: %s" %
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	315 datetime.datetime.fromtimestamp(record.process_time))
218 10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	316 if record.success:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	317 logging.info("Status: success")
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	318 else:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	319 logging.error("Status: failed")
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	320 logging.info("Entries:")
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	321 for entry in record.entries:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	322 rel_path = os.path.relpath(entry.path, ctx.app.root_dir)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	323 if pattern and not fnmatch.fnmatch(rel_path, pattern):
331 f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	324 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	325 if out_pattern and not (
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	326 any([o for o in entry.rel_outputs
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	327 if fnmatch.fnmatch(o, out_pattern)])):
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	328 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	329
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	330 flags = _get_flag_descriptions(
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	331 entry.flags,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	332 {
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	333 FLAG_PREPARED: 'prepared',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	334 FLAG_PROCESSED: 'processed',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	335 FLAG_BYPASSED_STRUCTURED_PROCESSING: 'external'})
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	336
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	337 logger.info(" - ")
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	338 logger.info(" path: %s" % rel_path)
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	339 logger.info(" out paths: %s" % entry.rel_outputs)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	340 logger.info(" flags: %s" % _join(flags))
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	341 logger.info(" proc tree: %s" % _format_proc_tree(
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	342 entry.proc_tree, 14*' '))
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	343
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	344 if entry.errors:
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	345 logger.error(" errors: %s" % entry.errors)
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	346
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	347
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	348 def _join(items, sep=', ', text_if_none='none'):
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	349 if items:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	350 return sep.join(items)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	351 return text_if_none
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	352
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	353
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	354 def _get_flag_descriptions(flags, descriptions):
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	355 res = []
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	356 for k, v in descriptions.items():
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	357 if flags & k:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	358 res.append(v)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	359 return res
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	360
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	361
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	362 def _format_proc_tree(tree, margin='', level=0):
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	363 name, children = tree
221 f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	364 res = '%s%s+ %s\n' % (margin if level > 0 else '', level * ' ', name)
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	365 if children:
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	366 for c in children:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	367 res += _format_proc_tree(c, margin, level + 1)
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	368 return res
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	369

Mercurial > piecrust2

annotate piecrust/commands/builtin/baking.py @ 666:81d9c3a3a0b5