Mercurial > piecrust2

annotate piecrust/commands/builtin/baking.py @ 380:f33712c4cfab

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
routing: Fix bugs with matching URLs with correct route but missing metadata. When matching a route like `/foo/%slug%` against an URL like `/foo`, the route will (correctly) return a match, but it will be completely missing the `slug` metadata, resulting in problems elsewhere. This change makes it so that any missing route metadata will be filled in with an empty string. And because this means generated URLs may differ from the incoming URL when using trailing slashes (`/foo/` _vs._ `/foo`), we make the assert in the chef server handle those discrepancies.
author Ludovic Chabant <ludovic@chabant.com>
date Sun, 10 May 2015 00:34:21 -0700
parents 2cd2b5d07129
children e7b865f8f335
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
121
300eb1c2cb14 Fix stupid bug.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
1 import time
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import logging
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import hashlib
92
0dd43c5f5484 More options for the `showrecord` command.
Ludovic Chabant <ludovic@chabant.com>
parents: 45
diff changeset
5 import fnmatch
0dd43c5f5484 More options for the `showrecord` command.
Ludovic Chabant <ludovic@chabant.com>
parents: 45
diff changeset
6 import datetime
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7 from piecrust.baking.baker import Baker
334
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
8 from piecrust.baking.records import (
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
9 BakeRecord, BakeRecordPageEntry, BakeRecordSubPageEntry)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 92
diff changeset
10 from piecrust.chefutil import format_timed
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11 from piecrust.commands.base import ChefCommand
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 92
diff changeset
12 from piecrust.processing.base import ProcessorPipeline
194
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
13 from piecrust.processing.records import (
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 220
diff changeset
14 ProcessorPipelineRecord,
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 220
diff changeset
15 FLAG_PREPARED, FLAG_PROCESSED, FLAG_OVERRIDEN,
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 220
diff changeset
16 FLAG_BYPASSED_STRUCTURED_PROCESSING)
334
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
17 from piecrust.rendering import PASS_FORMATTING, PASS_RENDERING
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 class BakeCommand(ChefCommand):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 super(BakeCommand, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 self.name = 'bake'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 self.description = "Bakes your website into static HTML files."
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29 def setupParser(self, parser, app):
192
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
30 parser.add_argument(
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
31 '-o', '--output',
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 help="The directory to put all the baked HTML files into "
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 "(defaults to `_counter`)")
192
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
34 parser.add_argument(
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
35 '-f', '--force',
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 help="Force re-baking the entire website.",
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 action='store_true')
192
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
38 parser.add_argument(
220
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical.
Ludovic Chabant <ludovic@chabant.com>
parents: 218
diff changeset
39 '--assets-only',
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical.
Ludovic Chabant <ludovic@chabant.com>
parents: 218
diff changeset
40 help="Only bake the assets (don't bake the web pages).",
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical.
Ludovic Chabant <ludovic@chabant.com>
parents: 218
diff changeset
41 action='store_true')
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical.
Ludovic Chabant <ludovic@chabant.com>
parents: 218
diff changeset
42 parser.add_argument(
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical.
Ludovic Chabant <ludovic@chabant.com>
parents: 218
diff changeset
43 '--html-only',
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical.
Ludovic Chabant <ludovic@chabant.com>
parents: 218
diff changeset
44 help="Only bake HTML files (don't run the asset pipeline).",
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 action='store_true')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 def run(self, ctx):
127
bc63dc20baa0 Fix how we pass the out directory to the baking modules.
Ludovic Chabant <ludovic@chabant.com>
parents: 121
diff changeset
48 out_dir = (ctx.args.output or
bc63dc20baa0 Fix how we pass the out directory to the baking modules.
Ludovic Chabant <ludovic@chabant.com>
parents: 121
diff changeset
49 os.path.join(ctx.app.root_dir, '_counter'))
bc63dc20baa0 Fix how we pass the out directory to the baking modules.
Ludovic Chabant <ludovic@chabant.com>
parents: 121
diff changeset
50
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
51 success = True
121
300eb1c2cb14 Fix stupid bug.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
52 start_time = time.clock()
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
53 try:
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 92
diff changeset
54 # Bake the site sources.
220
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical.
Ludovic Chabant <ludovic@chabant.com>
parents: 218
diff changeset
55 if not ctx.args.assets_only:
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical.
Ludovic Chabant <ludovic@chabant.com>
parents: 218
diff changeset
56 success = success & self._bakeSources(ctx, out_dir)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 92
diff changeset
57
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 92
diff changeset
58 # Bake the assets.
220
84e2bc2d16cb bake: Change arguments to selectively bake to make them symmetrical.
Ludovic Chabant <ludovic@chabant.com>
parents: 218
diff changeset
59 if not ctx.args.html_only:
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
60 success = success & self._bakeAssets(ctx, out_dir)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 92
diff changeset
61
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 92
diff changeset
62 # All done.
192
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
63 logger.info('-------------------------')
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
64 logger.info(format_timed(start_time, 'done baking'))
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
65 return 0 if success else 1
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
66 except Exception as ex:
43
f14889d6b067 Proper debug logging.
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
67 if ctx.app.debug:
f14889d6b067 Proper debug logging.
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
68 logger.exception(ex)
f14889d6b067 Proper debug logging.
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
69 else:
f14889d6b067 Proper debug logging.
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
70 logger.error(str(ex))
39
2f717f961996 Better error reporting and cache validation.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
71 return 1
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72
127
bc63dc20baa0 Fix how we pass the out directory to the baking modules.
Ludovic Chabant <ludovic@chabant.com>
parents: 121
diff changeset
73 def _bakeSources(self, ctx, out_dir):
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 92
diff changeset
74 baker = Baker(
127
bc63dc20baa0 Fix how we pass the out directory to the baking modules.
Ludovic Chabant <ludovic@chabant.com>
parents: 121
diff changeset
75 ctx.app, out_dir,
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
76 force=ctx.args.force)
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
77 record = baker.bake()
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
78 return record.success
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 92
diff changeset
79
127
bc63dc20baa0 Fix how we pass the out directory to the baking modules.
Ludovic Chabant <ludovic@chabant.com>
parents: 121
diff changeset
80 def _bakeAssets(self, ctx, out_dir):
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 92
diff changeset
81 proc = ProcessorPipeline(
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
82 ctx.app, out_dir,
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
83 force=ctx.args.force)
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
84 record = proc.run()
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
85 return record.success
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 92
diff changeset
86
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 92
diff changeset
87
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
88 class ShowRecordCommand(ChefCommand):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
89 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
90 super(ShowRecordCommand, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
91 self.name = 'showrecord'
192
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
92 self.description = ("Shows the bake record for a given output "
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
93 "directory.")
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
94
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
95 def setupParser(self, parser, app):
192
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
96 parser.add_argument(
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
97 '-o', '--output',
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98 help="The output directory for which to show the bake record "
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
99 "(defaults to `_counter`)",
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
100 nargs='?')
192
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
101 parser.add_argument(
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
102 '-p', '--path',
92
0dd43c5f5484 More options for the `showrecord` command.
Ludovic Chabant <ludovic@chabant.com>
parents: 45
diff changeset
103 help="A pattern that will be used to filter the relative path "
0dd43c5f5484 More options for the `showrecord` command.
Ludovic Chabant <ludovic@chabant.com>
parents: 45
diff changeset
104 "of entries to show.")
331
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
105 parser.add_argument(
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
106 '-t', '--out',
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
107 help="A pattern that will be used to filter the output path "
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
108 "of entries to show.")
334
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
109 parser.add_argument(
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
110 '--last',
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
111 type=int,
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
112 default=0,
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
113 help="Show the last Nth bake record.")
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
114
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
115 def run(self, ctx):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
116 out_dir = ctx.args.output or os.path.join(ctx.app.root_dir, '_counter')
334
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
117 record_id = hashlib.md5(out_dir.encode('utf8')).hexdigest()
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
118 suffix = '' if ctx.args.last == 0 else '.%d' % ctx.args.last
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
119 record_name = '%s%s.record' % (record_id, suffix)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
120
92
0dd43c5f5484 More options for the `showrecord` command.
Ludovic Chabant <ludovic@chabant.com>
parents: 45
diff changeset
121 pattern = None
0dd43c5f5484 More options for the `showrecord` command.
Ludovic Chabant <ludovic@chabant.com>
parents: 45
diff changeset
122 if ctx.args.path:
0dd43c5f5484 More options for the `showrecord` command.
Ludovic Chabant <ludovic@chabant.com>
parents: 45
diff changeset
123 pattern = '*%s*' % ctx.args.path.strip('*')
0dd43c5f5484 More options for the `showrecord` command.
Ludovic Chabant <ludovic@chabant.com>
parents: 45
diff changeset
124
331
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
125 out_pattern = None
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
126 if ctx.args.out:
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
127 out_pattern = '*%s*' % ctx.args.out.strip('*')
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
128
194
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
129 record_cache = ctx.app.cache.getCache('baker')
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
130 if not record_cache.has(record_name):
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
131 raise Exception("No record has been created for this output path. "
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
132 "Did you bake there yet?")
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
133
218
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
134 # Show the bake record.
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
135 record = BakeRecord.load(record_cache.getCachePath(record_name))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
136 logging.info("Bake record for: %s" % record.out_dir)
334
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
137 logging.info("From: %s" % record_name)
92
0dd43c5f5484 More options for the `showrecord` command.
Ludovic Chabant <ludovic@chabant.com>
parents: 45
diff changeset
138 logging.info("Last baked: %s" %
192
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
139 datetime.datetime.fromtimestamp(record.bake_time))
218
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
140 if record.success:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
141 logging.info("Status: success")
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
142 else:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
143 logging.error("Status: failed")
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
144 logging.info("Entries:")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
145 for entry in record.entries:
331
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
146 if pattern and not fnmatch.fnmatch(entry.rel_path, pattern):
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
147 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
148 if out_pattern and not (
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
149 any([o for o in entry.out_paths
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
150 if fnmatch.fnmatch(o, out_pattern)])):
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
151 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
152
334
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
153 flags = []
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
154 if entry.flags & BakeRecordPageEntry.FLAG_OVERRIDEN:
334
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
155 flags.append('overriden')
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
156
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
157 passes = {PASS_RENDERING: 'render', PASS_FORMATTING: 'format'}
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
158
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
159 logging.info(" - ")
192
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
160 logging.info(" path: %s" % entry.rel_path)
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
161 logging.info(" spec: %s:%s" % (entry.source_name,
4c0ab0b044fe cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 127
diff changeset
162 entry.rel_path))
334
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
163 if entry.taxonomy_info:
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
164 tn, t, sn = entry.taxonomy_info
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
165 logging.info(" taxonomy: %s (%s:%s)" %
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
166 (t, sn, tn))
334
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
167 else:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
168 logging.info(" taxonomy: <none>")
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 331
diff changeset
169 logging.info(" flags: %s" % ', '.join(flags))
92
0dd43c5f5484 More options for the `showrecord` command.
Ludovic Chabant <ludovic@chabant.com>
parents: 45
diff changeset
170 logging.info(" config: %s" % entry.config)
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
171
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
172 logging.info(" %d sub-pages:" % len(entry.subs))
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
173 for sub in entry.subs:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
174 logging.info(" - ")
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
175 logging.info(" URL: %s" % sub.out_uri)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
176 logging.info(" path: %s" % os.path.relpath(sub.out_path,
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
177 out_dir))
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
178 logging.info(" baked?: %s" % sub.was_baked)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
179
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
180 sub_flags = []
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
181 if sub.flags & BakeRecordSubPageEntry.FLAG_FORCED_BY_SOURCE:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
182 sub_flags.append('forced by source')
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
183 if sub.flags & BakeRecordSubPageEntry.FLAG_FORCED_BY_NO_PREVIOUS:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
184 sub_flags.append('forced by missing previous record entry')
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
185 if sub.flags & BakeRecordSubPageEntry.FLAG_FORCED_BY_PREVIOUS_ERRORS:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
186 sub_flags.append('forced by previous errors')
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
187 logging.info(" flags: %s" % ', '.join(sub_flags))
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
188
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
189 for p, pi in sub.render_passes.items():
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
190 logging.info(" %s pass:" % passes[p])
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
191 logging.info(" used srcs: %s" %
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
192 ', '.join(pi.used_source_names))
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
193 logging.info(" used terms: %s" %
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
194 ', '.join(
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
195 ['%s (%s:%s)' % (t, sn, tn)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
196 for sn, tn, t in pi.used_taxonomy_terms]))
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
197
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
198 if sub.errors:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
199 logging.error(" errors: %s" % sub.errors)
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
200
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
201 logging.info(" assets: %s" % ', '.join(entry.assets))
359
2cd2b5d07129 bake: Fix crash when handling bake errors.
Ludovic Chabant <ludovic@chabant.com>
parents: 338
diff changeset
202 if entry.errors:
2cd2b5d07129 bake: Fix crash when handling bake errors.
Ludovic Chabant <ludovic@chabant.com>
parents: 338
diff changeset
203 logging.error(" errors: %s" % entry.errors)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
204
194
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
205 record_cache = ctx.app.cache.getCache('proc')
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
206 if not record_cache.has(record_name):
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
207 return
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
208
218
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
209 # Show the pipeline record.
194
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
210 record = ProcessorPipelineRecord.load(
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
211 record_cache.getCachePath(record_name))
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
212 logging.info("")
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
213 logging.info("Processing record for: %s" % record.out_dir)
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
214 logging.info("Last baked: %s" %
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
215 datetime.datetime.fromtimestamp(record.process_time))
218
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
216 if record.success:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
217 logging.info("Status: success")
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
218 else:
10f24c62b05b showrecord: Show the overall status (success/failed) of the bake.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
219 logging.error("Status: failed")
194
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
220 logging.info("Entries:")
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
221 for entry in record.entries:
331
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
222 if pattern and not fnmatch.fnmatch(entry.rel_input, pattern):
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
223 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
224 if out_pattern and not (
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
225 any([o for o in entry.rel_outputs
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
226 if fnmatch.fnmatch(o, out_pattern)])):
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
227 continue
f0fc2a9d3191 showrecord: Add ability to filter on the output path.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
228
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 220
diff changeset
229 flags = []
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 220
diff changeset
230 if entry.flags & FLAG_PREPARED:
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 220
diff changeset
231 flags.append('prepared')
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 220
diff changeset
232 if entry.flags & FLAG_PROCESSED:
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 220
diff changeset
233 flags.append('processed')
194
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
234 if entry.flags & FLAG_OVERRIDEN:
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 220
diff changeset
235 flags.append('overriden')
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 220
diff changeset
236 if entry.flags & FLAG_BYPASSED_STRUCTURED_PROCESSING:
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 220
diff changeset
237 flags.append('external')
194
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
238 logger.info(" - ")
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
239 logger.info(" path: %s" % entry.rel_input)
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
240 logger.info(" out paths: %s" % entry.rel_outputs)
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
241 logger.info(" flags: %s" % flags)
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
242 logger.info(" proc tree: %s" % format_proc_tree(
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
243 entry.proc_tree, 14*' '))
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
244 if entry.errors:
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
245 logger.error(" errors: %s" % entry.errors)
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
246
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
247
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
248 def format_proc_tree(tree, margin='', level=0):
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
249 name, children = tree
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 220
diff changeset
250 res = '%s%s+ %s\n' % (margin if level > 0 else '', level * ' ', name)
194
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
251 if children:
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
252 for c in children:
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
253 res += format_proc_tree(c, margin, level + 1)
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
254 return res
5d8351cb32d8 showrecord: Also show the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 193
diff changeset
255