piecrust2: piecrust/commands/builtin/baking.py annotate

annotate piecrust/commands/builtin/baking.py @ 411:e7b865f8f335

bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.

author	Ludovic Chabant <ludovic@chabant.com>
date	Fri, 12 Jun 2015 17:09:19 -0700
parents	2cd2b5d07129
children	0e9a94b7fdfa

rev	line source
121 300eb1c2cb14 Fix stupid bug. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	1 import time
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os.path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import logging
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import hashlib
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	5 import fnmatch
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	6 import datetime
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	7 from piecrust.baking.baker import Baker
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	8 from piecrust.baking.records import (
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	9 BakeRecord, BakeRecordEntry, SubPageBakeInfo)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	10 from piecrust.chefutil import format_timed
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	11 from piecrust.commands.base import ChefCommand
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	12 from piecrust.processing.base import ProcessorPipeline
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	13 from piecrust.processing.records import (
221 f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	14 ProcessorPipelineRecord,
f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	15 FLAG_PREPARED, FLAG_PROCESSED, FLAG_OVERRIDEN,
f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	16 FLAG_BYPASSED_STRUCTURED_PROCESSING)
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	17 from piecrust.rendering import PASS_FORMATTING, PASS_RENDERING
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	23 class BakeCommand(ChefCommand):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	25 super(BakeCommand, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26 self.name = 'bake'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27 self.description = "Bakes your website into static HTML files."
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29 def setupParser(self, parser, app):
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	30 parser.add_argument(
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	31 '-o', '--output',
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32 help="The directory to put all the baked HTML files into "
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 "(defaults to `_counter`)")
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	34 parser.add_argument(
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	35 '-f', '--force',
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 help="Force re-baking the entire website.",
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37 action='store_true')
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	38 parser.add_argument(
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	39 '-w', '--workers',
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	40 help="The number of worker processes to spawn.",
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	41 type=int, default=-1)
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	42 parser.add_argument(
220 84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	43 '--assets-only',
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	44 help="Only bake the assets (don't bake the web pages).",
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	45 action='store_true')
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	46 parser.add_argument(
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	47 '--html-only',
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	48 help="Only bake HTML files (don't run the asset pipeline).",
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 action='store_true')
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	50 parser.add_argument(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	51 '--show-timers',
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	52 help="Show detailed timing information.",
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	53 action='store_true')
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	55 def run(self, ctx):
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	56 out_dir = (ctx.args.output or
bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	57 os.path.join(ctx.app.root_dir, '_counter'))
bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	58
215 a47580a0955b bake: Better error handling for the processing pipeline. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	59 success = True
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	60 start_time = time.perf_counter()
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	61 try:
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	62 # Bake the site sources.
220 84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	63 if not ctx.args.assets_only:
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	64 success = success & self._bakeSources(ctx, out_dir)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	65
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	66 # Bake the assets.
220 84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical. Ludovic Chabant <ludovic@chabant.com> parents: 218 diff changeset	67 if not ctx.args.html_only:
215 a47580a0955b bake: Better error handling for the processing pipeline. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	68 success = success & self._bakeAssets(ctx, out_dir)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	69
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	70 # All done.
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	71 logger.info('-------------------------')
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	72 logger.info(format_timed(start_time, 'done baking'))
215 a47580a0955b bake: Better error handling for the processing pipeline. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	73 return 0 if success else 1
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	74 except Exception as ex:
43 f14889d6b067 Proper debug logging. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	75 if ctx.app.debug:
f14889d6b067 Proper debug logging. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	76 logger.exception(ex)
f14889d6b067 Proper debug logging. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	77 else:
f14889d6b067 Proper debug logging. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	78 logger.error(str(ex))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	79 return 1
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	81 def _bakeSources(self, ctx, out_dir):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	82 if ctx.args.workers > 0:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	83 ctx.app.config.set('baker/workers', ctx.args.workers)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	84 baker = Baker(
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	85 ctx.app, out_dir,
205 e725af1d48fb bake: Changes in how assets directories are configured. Ludovic Chabant <ludovic@chabant.com> parents: 201 diff changeset	86 force=ctx.args.force)
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 215 diff changeset	87 record = baker.bake()
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	88
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	89 if ctx.args.show_timers:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	90 if record.timers:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	91 from colorama import Fore
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	92 logger.info("-------------------")
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	93 logger.info("Timing information:")
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	94 for name in sorted(record.timers.keys()):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	95 val_str = '%8.1f s' % record.timers[name]
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	96 logger.info(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	97 "[%s%s%s] %s" %
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	98 (Fore.GREEN, val_str, Fore.RESET, name))
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	99 else:
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	100 logger.warning("Timing information is not available.")
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	101
217 1f4c3dae1fe8 bake: Better error handling for site baking. Ludovic Chabant <ludovic@chabant.com> parents: 215 diff changeset	102 return record.success
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	103
127 bc63dc20baa0 Fix how we pass the out directory to the baking modules. Ludovic Chabant <ludovic@chabant.com> parents: 121 diff changeset	104 def _bakeAssets(self, ctx, out_dir):
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	105 proc = ProcessorPipeline(
205 e725af1d48fb bake: Changes in how assets directories are configured. Ludovic Chabant <ludovic@chabant.com> parents: 201 diff changeset	106 ctx.app, out_dir,
e725af1d48fb bake: Changes in how assets directories are configured. Ludovic Chabant <ludovic@chabant.com> parents: 201 diff changeset	107 force=ctx.args.force)
215 a47580a0955b bake: Better error handling for the processing pipeline. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	108 record = proc.run()
a47580a0955b bake: Better error handling for the processing pipeline. Ludovic Chabant <ludovic@chabant.com> parents: 205 diff changeset	109 return record.success
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	110
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 92 diff changeset	111
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	112 class ShowRecordCommand(ChefCommand):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	113 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	114 super(ShowRecordCommand, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	115 self.name = 'showrecord'
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	116 self.description = ("Shows the bake record for a given output "
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	117 "directory.")
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	118
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	119 def setupParser(self, parser, app):
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	120 parser.add_argument(
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	121 '-o', '--output',
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	122 help="The output directory for which to show the bake record "
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	123 "(defaults to `_counter`)",
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	124 nargs='?')
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	125 parser.add_argument(
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	126 '-p', '--path',
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	127 help="A pattern that will be used to filter the relative path "
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	128 "of entries to show.")
331 f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	129 parser.add_argument(
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	130 '-t', '--out',
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	131 help="A pattern that will be used to filter the output path "
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	132 "of entries to show.")
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	133 parser.add_argument(
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	134 '--last',
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	135 type=int,
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	136 default=0,
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	137 help="Show the last Nth bake record.")
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	138
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	139 def run(self, ctx):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	140 out_dir = ctx.args.output or os.path.join(ctx.app.root_dir, '_counter')
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	141 record_id = hashlib.md5(out_dir.encode('utf8')).hexdigest()
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	142 suffix = '' if ctx.args.last == 0 else '.%d' % ctx.args.last
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	143 record_name = '%s%s.record' % (record_id, suffix)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	144
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	145 pattern = None
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	146 if ctx.args.path:
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	147 pattern = '%s' % ctx.args.path.strip('*')
0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	148
331 f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	149 out_pattern = None
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	150 if ctx.args.out:
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	151 out_pattern = '%s' % ctx.args.out.strip('*')
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	152
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	153 record_cache = ctx.app.cache.getCache('baker')
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	154 if not record_cache.has(record_name):
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	155 raise Exception("No record has been created for this output path. "
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	156 "Did you bake there yet?")
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	157
218 10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	158 # Show the bake record.
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	159 record = BakeRecord.load(record_cache.getCachePath(record_name))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	160 logging.info("Bake record for: %s" % record.out_dir)
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	161 logging.info("From: %s" % record_name)
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	162 logging.info("Last baked: %s" %
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	163 datetime.datetime.fromtimestamp(record.bake_time))
218 10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	164 if record.success:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	165 logging.info("Status: success")
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	166 else:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	167 logging.error("Status: failed")
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	168 logging.info("Entries:")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	169 for entry in record.entries:
331 f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	170 if pattern and not fnmatch.fnmatch(entry.rel_path, pattern):
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	171 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	172 if out_pattern and not (
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	173 any([o for o in entry.out_paths
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	174 if fnmatch.fnmatch(o, out_pattern)])):
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	175 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	176
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	177 flags = []
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	178 if entry.flags & BakeRecordEntry.FLAG_OVERRIDEN:
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	179 flags.append('overriden')
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	180
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	181 passes = {PASS_RENDERING: 'render', PASS_FORMATTING: 'format'}
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	182
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	183 logging.info(" - ")
192 4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	184 logging.info(" path: %s" % entry.rel_path)
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	185 logging.info(" spec: %s:%s" % (entry.source_name,
4c0ab0b044fe cosmetic: Fix some PEP8 issues. Ludovic Chabant <ludovic@chabant.com> parents: 127 diff changeset	186 entry.rel_path))
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	187 if entry.taxonomy_info:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	188 tax_name, term, source_name = entry.taxonomy_info
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	189 logging.info(" taxonomy: %s (%s:%s)" %
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	190 (term, source_name, tax_name))
334 b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	191 else:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	192 logging.info(" taxonomy: <none>")
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes. Ludovic Chabant <ludovic@chabant.com> parents: 331 diff changeset	193 logging.info(" flags: %s" % ', '.join(flags))
92 0dd43c5f5484 More options for the `showrecord` command. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	194 logging.info(" config: %s" % entry.config)
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	195
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	196 logging.info(" %d sub-pages:" % len(entry.subs))
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	197 for sub in entry.subs:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	198 logging.info(" - ")
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	199 logging.info(" URL: %s" % sub.out_uri)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	200 logging.info(" path: %s" % os.path.relpath(sub.out_path,
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	201 out_dir))
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	202 logging.info(" baked?: %s" % sub.was_baked)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	203
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	204 sub_flags = []
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	205 if sub.flags & SubPageBakeInfo.FLAG_FORCED_BY_SOURCE:
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	206 sub_flags.append('forced by source')
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	207 if sub.flags & SubPageBakeInfo.FLAG_FORCED_BY_NO_PREVIOUS:
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	208 sub_flags.append('forced by missing previous record entry')
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	209 if sub.flags & SubPageBakeInfo.FLAG_FORCED_BY_PREVIOUS_ERRORS:
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	210 sub_flags.append('forced by previous errors')
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	211 logging.info(" flags: %s" % ', '.join(sub_flags))
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	212
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	213 for p, pi in sub.render_passes.items():
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	214 logging.info(" %s pass:" % passes[p])
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	215 logging.info(" used srcs: %s" %
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	216 ', '.join(pi.used_source_names))
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	217 logging.info(" used terms: %s" %
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	218 ', '.join(
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	219 ['%s (%s:%s)' % (t, sn, tn)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	220 for sn, tn, t in
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 359 diff changeset	221 pi.used_taxonomy_terms]))
338 938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	222
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	223 if sub.errors:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	224 logging.error(" errors: %s" % sub.errors)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	225
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs. Ludovic Chabant <ludovic@chabant.com> parents: 334 diff changeset	226 logging.info(" assets: %s" % ', '.join(entry.assets))
359 2cd2b5d07129 bake: Fix crash when handling bake errors. Ludovic Chabant <ludovic@chabant.com> parents: 338 diff changeset	227 if entry.errors:
2cd2b5d07129 bake: Fix crash when handling bake errors. Ludovic Chabant <ludovic@chabant.com> parents: 338 diff changeset	228 logging.error(" errors: %s" % entry.errors)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	229
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	230 record_cache = ctx.app.cache.getCache('proc')
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	231 if not record_cache.has(record_name):
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	232 return
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	233
218 10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	234 # Show the pipeline record.
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	235 record = ProcessorPipelineRecord.load(
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	236 record_cache.getCachePath(record_name))
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	237 logging.info("")
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	238 logging.info("Processing record for: %s" % record.out_dir)
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	239 logging.info("Last baked: %s" %
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	240 datetime.datetime.fromtimestamp(record.process_time))
218 10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	241 if record.success:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	242 logging.info("Status: success")
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	243 else:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake. Ludovic Chabant <ludovic@chabant.com> parents: 217 diff changeset	244 logging.error("Status: failed")
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	245 logging.info("Entries:")
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	246 for entry in record.entries:
331 f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	247 if pattern and not fnmatch.fnmatch(entry.rel_input, pattern):
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	248 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	249 if out_pattern and not (
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	250 any([o for o in entry.rel_outputs
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	251 if fnmatch.fnmatch(o, out_pattern)])):
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	252 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path. Ludovic Chabant <ludovic@chabant.com> parents: 221 diff changeset	253
221 f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	254 flags = []
f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	255 if entry.flags & FLAG_PREPARED:
f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	256 flags.append('prepared')
f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	257 if entry.flags & FLAG_PROCESSED:
f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	258 flags.append('processed')
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	259 if entry.flags & FLAG_OVERRIDEN:
221 f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	260 flags.append('overriden')
f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	261 if entry.flags & FLAG_BYPASSED_STRUCTURED_PROCESSING:
f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	262 flags.append('external')
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	263 logger.info(" - ")
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	264 logger.info(" path: %s" % entry.rel_input)
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	265 logger.info(" out paths: %s" % entry.rel_outputs)
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	266 logger.info(" flags: %s" % flags)
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	267 logger.info(" proc tree: %s" % format_proc_tree(
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	268 entry.proc_tree, 14*' '))
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	269 if entry.errors:
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	270 logger.error(" errors: %s" % entry.errors)
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	271
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	272
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	273 def format_proc_tree(tree, margin='', level=0):
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	274 name, children = tree
221 f82262f59600 bake: Fix processing record bugs and error logging for external processes. Ludovic Chabant <ludovic@chabant.com> parents: 220 diff changeset	275 res = '%s%s+ %s\n' % (margin if level > 0 else '', level * ' ', name)
194 5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	276 if children:
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	277 for c in children:
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	278 res += format_proc_tree(c, margin, level + 1)
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	279 return res
5d8351cb32d8 showrecord: Also show the pipeline record. Ludovic Chabant <ludovic@chabant.com> parents: 193 diff changeset	280

Mercurial > piecrust2

annotate piecrust/commands/builtin/baking.py @ 411:e7b865f8f335