piecrust2: piecrust/commands/builtin/baking.py annotate

annotate piecrust/commands/builtin/baking.py @ 852:4850f8c21b6e

core: Start of the big refactor for PieCrust 3.0. * Everything is a `ContentSource`, including assets directories. * Most content sources are subclasses of the base file-system source. * A source is processed by a "pipeline", and there are 2 built-in pipelines, one for assets and one for pages. The asset pipeline is vaguely functional, but the page pipeline is completely broken right now. * Rewrite the baking process as just running appropriate pipelines on each content item. This should allow for better parallelization.

author	Ludovic Chabant <ludovic@chabant.com>
date	Wed, 17 May 2017 00:11:48 -0700
parents	08e6484a2600
children	f070a4fc033c

rev	line source
121 300eb1c2cb14 Fix stupid bug. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	1 import time
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os.path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import logging
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import hashlib
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	5 import fnmatch
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	6 import datetime
421 4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	7 from colorama import Fore
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8 from piecrust.commands.base import ChefCommand
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	9
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	10
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	11 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	14 class BakeCommand(ChefCommand):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	15 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16 super(BakeCommand, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17 self.name = 'bake'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18 self.description = "Bakes your website into static HTML files."
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20 def setupParser(self, parser, app):
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	21 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	22 '-o', '--output',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	23 help="The directory to put all the baked HTML files into "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	24 "(defaults to `_counter`)")
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	25 parser.add_argument(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	26 '-f', '--force',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	27 help="Force re-baking the entire website.",
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	28 action='store_true')
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	29 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	30 '-p', '--pipelines',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	31 help="The pipelines to run.",
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	32 nargs='*')
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	33 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	34 '-w', '--workers',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	35 help="The number of worker processes to spawn.",
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	36 type=int, default=-1)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	37 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	38 '--batch-size',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	39 help="The number of jobs per batch.",
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	40 type=int, default=-1)
462 04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 421 diff changeset	41 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	42 '--assets-only',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	43 help="Only bake the assets (don't bake the web pages).",
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	44 action='store_true')
220 84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	45 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	46 '--html-only',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	47 help="Only bake the pages (don't run the asset pipeline).",
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	48 action='store_true')
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	49 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	50 '--show-stats',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	51 help="Show detailed information about the bake.",
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	52 action='store_true')
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54 def run(self, ctx):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	55 from piecrust.chefutil import format_timed
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	56
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	57 out_dir = (ctx.args.output or
bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	58 os.path.join(ctx.app.root_dir, '_counter'))
bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	59
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	60 start_time = time.perf_counter()
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	61 try:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	62 records = self._doBake(ctx, out_dir)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	63
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	64 # Show merged stats.
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	65 if ctx.args.show_stats:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	66 logger.info("-------------------")
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	67 logger.info("Timing information:")
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	68 _show_stats(records.stats)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	69
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	70 # All done.
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	71 logger.info('-------------------------')
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	72 logger.info(format_timed(start_time, 'done baking'))
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	73 return 0 if records.success else 1
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	74 except Exception as ex:
43 f14889d6b067 Proper debug logging. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	75 if ctx.app.debug:
f14889d6b067 Proper debug logging. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	76 logger.exception(ex)
f14889d6b067 Proper debug logging. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	77 else:
f14889d6b067 Proper debug logging. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	78 logger.error(str(ex))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	79 return 1
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	81 def _doBake(self, ctx, out_dir):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	82 from piecrust.baking.baker import Baker
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	83
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	84 if ctx.args.workers > 0:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	85 ctx.app.config.set('baker/workers', ctx.args.workers)
462 04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 421 diff changeset	86 if ctx.args.batch_size > 0:
04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 421 diff changeset	87 ctx.app.config.set('baker/batch_size', ctx.args.batch_size)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	88
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	89 allowed_pipelines = None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	90 if ctx.args.html_only:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	91 allowed_pipelines = ['page']
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	92 elif ctx.args.assets_only:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	93 allowed_pipelines = ['asset']
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	94 elif ctx.args.pipelines:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	95 allowed_pipelines = ctx.args.pipelines
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	96
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	97 baker = Baker(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	98 ctx.appfactory, ctx.app, out_dir,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	99 force=ctx.args.force,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	100 allowed_pipelines=allowed_pipelines)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	101 records = baker.bake()
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	102
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	103 return records
421 4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	104
4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	105
692 c11a4339fccb bake: Show more stats. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	106 def _show_stats(stats, *, full=False):
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	107 indent = ' '
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	108
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	109 logger.info(' Timers:')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	110 for name, val in sorted(stats.timers.items(), key=lambda i: i[1],
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	111 reverse=True):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	112 val_str = '%8.1f s' % val
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	113 logger.info(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	114 "%s[%s%s%s] %s" %
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	115 (indent, Fore.GREEN, val_str, Fore.RESET, name))
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	116
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	117 logger.info(' Counters:')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	118 for name in sorted(stats.counters.keys()):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	119 val_str = '%8d ' % stats.counters[name]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	120 logger.info(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	121 "%s[%s%s%s] %s" %
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	122 (indent, Fore.GREEN, val_str, Fore.RESET, name))
421 4a43d7015b75 bake: Improve performance timers reports. Ludovic Chabant <ludovic@chabant.com> parents: 415 diff changeset	123
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	124 logger.info(' Manifests:')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	125 for name in sorted(stats.manifests.keys()):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	126 val = stats.manifests[name]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	127 logger.info(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	128 "%s[%s%s%s] [%d entries]" %
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	129 (indent, Fore.CYAN, name, Fore.RESET, len(val)))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	130 if full:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	131 for v in val:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	132 logger.info("%s - %s" % (indent, v))
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	133
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	134
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	135 class ShowRecordCommand(ChefCommand):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	136 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	137 super(ShowRecordCommand, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	138 self.name = 'showrecord'
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	139 self.description = ("Shows the bake record for a given output "
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	140 "directory.")
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	141
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	142 def setupParser(self, parser, app):
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	143 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	144 '-o', '--output',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	145 help="The output directory for which to show the bake record "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	146 "(defaults to `_counter`)",
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	147 nargs='?')
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	148 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	149 '-p', '--path',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	150 help="A pattern that will be used to filter the relative path "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	151 "of entries to show.")
331 f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	152 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	153 '-t', '--out',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	154 help="A pattern that will be used to filter the output path "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	155 "of entries to show.")
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	156 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	157 '--last',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	158 type=int,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	159 default=0,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	160 help="Show the last Nth bake record.")
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	161 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	162 '--html-only',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	163 action='store_true',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	164 help="Only show records for pages (not from the asset "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	165 "pipeline).")
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	166 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	167 '--assets-only',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	168 action='store_true',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	169 help="Only show records for assets (not from pages).")
692 c11a4339fccb bake: Show more stats. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	170 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	171 '--show-stats',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	172 action='store_true',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	173 help="Show stats from the record.")
801 87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	174 parser.add_argument(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	175 '--show-manifest',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	176 help="Show manifest entries from the record.")
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	177
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	178 def run(self, ctx):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	179 from piecrust.processing.records import (
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	180 FLAG_PREPARED, FLAG_PROCESSED, FLAG_BYPASSED_STRUCTURED_PROCESSING,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	181 FLAG_COLLAPSED_FROM_LAST_RUN)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	182 from piecrust.rendering import (
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	183 PASS_FORMATTING, PASS_RENDERING)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	184
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	185 out_dir = ctx.args.output or os.path.join(ctx.app.root_dir, '_counter')
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	186 record_id = hashlib.md5(out_dir.encode('utf8')).hexdigest()
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	187 suffix = '' if ctx.args.last == 0 else '.%d' % ctx.args.last
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	188 record_name = '%s%s.record' % (record_id, suffix)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	189
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	190 pattern = None
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	191 if ctx.args.path:
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	192 pattern = '%s' % ctx.args.path.strip('*')
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	193
331 f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	194 out_pattern = None
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	195 if ctx.args.out:
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	196 out_pattern = '%s' % ctx.args.out.strip('*')
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	197
801 87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	198 if not ctx.args.show_stats and not ctx.args.show_manifest:
701 066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	199 if not ctx.args.assets_only:
066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	200 self._showBakeRecord(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	201 ctx, record_name, pattern, out_pattern)
701 066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	202 if not ctx.args.html_only:
066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	203 self._showProcessingRecord(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	204 ctx, record_name, pattern, out_pattern)
801 87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	205 return
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	206
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	207 stats = {}
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	208 bake_rec = self._getBakeRecord(ctx, record_name)
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	209 if bake_rec:
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	210 _merge_stats(bake_rec.stats, stats)
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	211 proc_rec = self._getProcessingRecord(ctx, record_name)
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	212 if proc_rec:
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	213 _merge_stats(proc_rec.stats, stats)
692 c11a4339fccb bake: Show more stats. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	214
c11a4339fccb bake: Show more stats. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	215 if ctx.args.show_stats:
701 066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	216 _show_stats(stats, full=False)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	217
801 87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	218 if ctx.args.show_manifest:
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	219 for name in sorted(stats.keys()):
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	220 logger.info('%s:' % name)
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	221 s = stats[name]
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	222 for name in sorted(s.manifests.keys()):
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	223 if ctx.args.show_manifest.lower() in name.lower():
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	224 val = s.manifests[name]
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	225 logger.info(
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	226 " [%s%s%s] [%d entries]" %
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	227 (Fore.CYAN, name, Fore.RESET, len(val)))
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	228 for v in val:
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	229 logger.info(" - %s" % v)
87df68be4cc8 showrecord: Add `show-manifest` argument. Ludovic Chabant <ludovic@chabant.com> parents: 754 diff changeset	230
701 066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	231 def _getBakeRecord(self, ctx, record_name):
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	232 record_cache = ctx.app.cache.getCache('baker')
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	233 if not record_cache.has(record_name):
698 33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	234 logger.warning(
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	235 "No page bake record has been created for this output "
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	236 "path.")
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	237 return None
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	238
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	239 record = BakeRecord.load(record_cache.getCachePath(record_name))
701 066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	240 return record
066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	241
066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	242 def _showBakeRecord(self, ctx, record_name, pattern, out_pattern):
066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	243 record = self._getBakeRecord(ctx, record_name)
754 78de21447e4e showrecord: Fix some crashes and incorrect information. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	244 if record is None:
78de21447e4e showrecord: Fix some crashes and incorrect information. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	245 return
701 066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	246
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	247 logging.info("Bake record for: %s" % record.out_dir)
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	248 logging.info("From: %s" % record_name)
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	249 logging.info("Last baked: %s" %
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	250 datetime.datetime.fromtimestamp(record.bake_time))
218 10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	251 if record.success:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	252 logging.info("Status: success")
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	253 else:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	254 logging.error("Status: failed")
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	255 logging.info("Entries:")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	256 for entry in record.entries:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	257 if pattern and not fnmatch.fnmatch(entry.path, pattern):
331 f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	258 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	259 if out_pattern and not (
754 78de21447e4e showrecord: Fix some crashes and incorrect information. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	260 any([o for o in entry.all_out_paths
331 f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	261 if fnmatch.fnmatch(o, out_pattern)])):
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	262 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	263
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	264 flags = _get_flag_descriptions(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	265 entry.flags,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	266 {
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	267 BakeRecordEntry.FLAG_NEW: 'new',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	268 BakeRecordEntry.FLAG_SOURCE_MODIFIED: 'modified',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	269 BakeRecordEntry.FLAG_OVERRIDEN: 'overriden'})
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	270
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	271 logging.info(" - ")
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	272
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	273 rel_path = os.path.relpath(entry.path, ctx.app.root_dir)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	274 logging.info(" path: %s" % rel_path)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	275 logging.info(" source: %s" % entry.source_name)
711 ab5c6a8ae90a bake: Replace hard-coded taxonomy support with "generator" system. Ludovic Chabant <ludovic@chabant.com> parents: 701 diff changeset	276 if entry.extra_key:
ab5c6a8ae90a bake: Replace hard-coded taxonomy support with "generator" system. Ludovic Chabant <ludovic@chabant.com> parents: 701 diff changeset	277 logging.info(" extra key: %s" % entry.extra_key)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	278 logging.info(" flags: %s" % _join(flags))
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	279 logging.info(" config: %s" % entry.config)
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	280
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	281 if entry.errors:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	282 logging.error(" errors: %s" % entry.errors)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	283
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	284 logging.info(" %d sub-pages:" % len(entry.subs))
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	285 for sub in entry.subs:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	286 sub_flags = _get_flag_descriptions(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	287 sub.flags,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	288 {
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	289 SubPageBakeInfo.FLAG_BAKED: 'baked',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	290 SubPageBakeInfo.FLAG_FORCED_BY_SOURCE:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	291 'forced by source',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	292 SubPageBakeInfo.FLAG_FORCED_BY_NO_PREVIOUS:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	293 'forced by missing previous record entry',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	294 SubPageBakeInfo.FLAG_FORCED_BY_PREVIOUS_ERRORS:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	295 'forced by previous errors',
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	296 SubPageBakeInfo.FLAG_FORMATTING_INVALIDATED:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 804 diff changeset	297 'formatting invalidated'})
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	298
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	299 logging.info(" - ")
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	300 logging.info(" URL: %s" % sub.out_uri)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	301 logging.info(" path: %s" % os.path.relpath(
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	302 sub.out_path, record.out_dir))
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	303 logging.info(" flags: %s" % _join(sub_flags))
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	304
698 33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	305 pass_names = {
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	306 PASS_FORMATTING: 'formatting pass',
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	307 PASS_RENDERING: 'rendering pass'}
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	308 for p, ri in enumerate(sub.render_info):
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	309 logging.info(" - %s" % pass_names[p])
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	310 if not ri:
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	311 logging.info(" no info")
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	312 continue
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	313
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	314 logging.info(" used sources: %s" %
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	315 _join(ri.used_source_names))
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	316 pgn_info = 'no'
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	317 if ri.used_pagination:
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	318 pgn_info = 'yes'
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	319 if ri.pagination_has_more:
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	320 pgn_info += ', has more'
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	321 logging.info(" used pagination: %s", pgn_info)
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	322 logging.info(" used assets: %s",
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	323 'yes' if ri.used_assets else 'no')
711 ab5c6a8ae90a bake: Replace hard-coded taxonomy support with "generator" system. Ludovic Chabant <ludovic@chabant.com> parents: 701 diff changeset	324 logging.info(" other info:")
ab5c6a8ae90a bake: Replace hard-coded taxonomy support with "generator" system. Ludovic Chabant <ludovic@chabant.com> parents: 701 diff changeset	325 for k, v in ri._custom_info.items():
ab5c6a8ae90a bake: Replace hard-coded taxonomy support with "generator" system. Ludovic Chabant <ludovic@chabant.com> parents: 701 diff changeset	326 logging.info(" - %s: %s" % (k, v))
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	327
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	328 if sub.errors:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	329 logging.error(" errors: %s" % sub.errors)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	330
701 066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	331 def _getProcessingRecord(self, ctx, record_name):
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	332 record_cache = ctx.app.cache.getCache('proc')
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	333 if not record_cache.has(record_name):
698 33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	334 logger.warning(
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	335 "No asset processing record has been created for this "
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	336 "output path.")
33ab9badfd7a render: Change how we store render passes info. Ludovic Chabant <ludovic@chabant.com> parents: 692 diff changeset	337 return None
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	338
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	339 record = ProcessorPipelineRecord.load(
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	340 record_cache.getCachePath(record_name))
701 066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	341 return record
066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	342
066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	343 def _showProcessingRecord(self, ctx, record_name, pattern, out_pattern):
066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	344 record = self._getProcessingRecord(ctx, record_name)
066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	345 if record is None:
066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	346 return
066d6156525c showrecord: Don't print the record when you just want the stats. Ludovic Chabant <ludovic@chabant.com> parents: 698 diff changeset	347
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	348 logging.info("")
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	349 logging.info("Processing record for: %s" % record.out_dir)
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	350 logging.info("Last baked: %s" %
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	351 datetime.datetime.fromtimestamp(record.process_time))
218 10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	352 if record.success:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	353 logging.info("Status: success")
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	354 else:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	355 logging.error("Status: failed")
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	356 logging.info("Entries:")
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	357 for entry in record.entries:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	358 rel_path = os.path.relpath(entry.path, ctx.app.root_dir)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	359 if pattern and not fnmatch.fnmatch(rel_path, pattern):
331 f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	360 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	361 if out_pattern and not (
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	362 any([o for o in entry.rel_outputs
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	363 if fnmatch.fnmatch(o, out_pattern)])):
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	364 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	365
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	366 flags = _get_flag_descriptions(
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	367 entry.flags,
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	368 {
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	369 FLAG_PREPARED: 'prepared',
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	370 FLAG_PROCESSED: 'processed',
754 78de21447e4e showrecord: Fix some crashes and incorrect information. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	371 FLAG_BYPASSED_STRUCTURED_PROCESSING: 'external',
78de21447e4e showrecord: Fix some crashes and incorrect information. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	372 FLAG_COLLAPSED_FROM_LAST_RUN: 'from last run'})
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	373
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	374 logger.info(" - ")
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	375 logger.info(" path: %s" % rel_path)
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	376 logger.info(" out paths: %s" % entry.rel_outputs)
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	377 logger.info(" flags: %s" % _join(flags))
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	378 logger.info(" proc tree: %s" % _format_proc_tree(
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	379 entry.proc_tree, 14*' '))
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	380
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	381 if entry.errors:
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	382 logger.error(" errors: %s" % entry.errors)
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	383
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	384
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	385 def _join(items, sep=', ', text_if_none='none'):
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	386 if items:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	387 return sep.join(items)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	388 return text_if_none
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	389
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	390
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	391 def _get_flag_descriptions(flags, descriptions):
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	392 res = []
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	393 for k, v in descriptions.items():
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	394 if flags & k:
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	395 res.append(v)
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	396 return res
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	397
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	398
0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	399 def _format_proc_tree(tree, margin='', level=0):
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	400 name, children = tree
221 f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	401 res = '%s%s+ %s\n' % (margin if level > 0 else '', level * ' ', name)
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	402 if children:
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	403 for c in children:
415 0e9a94b7fdfa bake: Improve bake record information. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	404 res += _format_proc_tree(c, margin, level + 1)
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	405 return res
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	406

Mercurial > piecrust2

annotate piecrust/commands/builtin/baking.py @ 852:4850f8c21b6e