Mercurial > piecrust2
annotate piecrust/baking/records.py @ 415:0e9a94b7fdfa
bake: Improve bake record information.
* Store things in the bake record that require less interaction between the
master process and the workers. For instance, don't store the paginator
object in the render pass info -- instead, just store whether pagination
was used, and whether it had more items.
* Simplify information passing between workers and bake passes by saving the
rendering info to the JSON cache. This means the "render first sub" job
doesn't have to return anything except errors now.
* Add more performance counter info.
author | Ludovic Chabant <ludovic@chabant.com> |
---|---|
date | Sat, 20 Jun 2015 19:23:16 -0700 |
parents | e7b865f8f335 |
children | 61d606fbc313 |
rev | line source |
---|---|
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
1 import copy |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
2 import os.path |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
3 import hashlib |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
4 import logging |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
5 from piecrust.records import Record, TransitionalRecord |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
6 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
7 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
8 logger = logging.getLogger(__name__) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
9 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
10 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
11 def _get_transition_key(path, taxonomy_info=None): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
12 key = path |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
13 if taxonomy_info: |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
14 key += '+%s:%s=' % (taxonomy_info.source_name, |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
15 taxonomy_info.taxonomy_name) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
16 if isinstance(taxonomy_info.term, tuple): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
17 key += '/'.join(taxonomy_info.term) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
18 else: |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
19 key += taxonomy_info.term |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
20 return hashlib.md5(key.encode('utf8')).hexdigest() |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
21 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
22 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
23 class BakeRecord(Record): |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
24 RECORD_VERSION = 14 |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
25 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
26 def __init__(self): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
27 super(BakeRecord, self).__init__() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
28 self.out_dir = None |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
29 self.bake_time = None |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
30 self.baked_count = {} |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
31 self.timers = None |
217
1f4c3dae1fe8
bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
120
diff
changeset
|
32 self.success = True |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
33 |
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
34 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
35 class SubPageBakeInfo(object): |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
36 FLAG_NONE = 0 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
37 FLAG_BAKED = 2**0 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
38 FLAG_FORCED_BY_SOURCE = 2**1 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
39 FLAG_FORCED_BY_NO_PREVIOUS = 2**2 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
40 FLAG_FORCED_BY_PREVIOUS_ERRORS = 2**3 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
41 FLAG_FORMATTING_INVALIDATED = 2**4 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
42 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
43 def __init__(self, out_uri, out_path): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
44 self.out_uri = out_uri |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
45 self.out_path = out_path |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
46 self.flags = self.FLAG_NONE |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
47 self.errors = [] |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
48 self.render_info = None |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
49 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
50 @property |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
51 def was_clean(self): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
52 return (self.flags & self.FLAG_BAKED) == 0 and len(self.errors) == 0 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
53 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
54 @property |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
55 def was_baked(self): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
56 return (self.flags & self.FLAG_BAKED) != 0 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
57 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
58 @property |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
59 def was_baked_successfully(self): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
60 return self.was_baked and len(self.errors) == 0 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
61 |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
62 def anyPass(self, func): |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
63 assert self.render_info is not None |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
64 for p, pinfo in self.render_info.items(): |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
65 if func(pinfo): |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
66 return True |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
67 return False |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
68 |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
69 def copyRenderInfo(self): |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
70 assert self.render_info |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
71 return copy.deepcopy(self.render_info) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
72 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
73 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
74 class TaxonomyInfo(object): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
75 def __init__(self, taxonomy_name, source_name, term): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
76 self.taxonomy_name = taxonomy_name |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
77 self.source_name = source_name |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
78 self.term = term |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
79 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
80 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
81 class BakeRecordEntry(object): |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
82 """ An entry in the bake record. |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
83 |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
84 The `taxonomy_info` attribute should be a tuple of the form: |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
85 (taxonomy name, term, source name) |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
86 """ |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
87 FLAG_NONE = 0 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
88 FLAG_NEW = 2**0 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
89 FLAG_SOURCE_MODIFIED = 2**1 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
90 FLAG_OVERRIDEN = 2**2 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
91 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
92 def __init__(self, source_name, path, taxonomy_info=None): |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
93 self.source_name = source_name |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
94 self.path = path |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
95 self.taxonomy_info = taxonomy_info |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
96 self.flags = self.FLAG_NONE |
50
2fec3ee1298f
Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents:
23
diff
changeset
|
97 self.config = None |
359
2cd2b5d07129
bake: Fix crash when handling bake errors.
Ludovic Chabant <ludovic@chabant.com>
parents:
338
diff
changeset
|
98 self.errors = [] |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
99 self.subs = [] |
334
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
100 |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
101 @property |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
102 def path_mtime(self): |
b034f6f15e22
bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents:
217
diff
changeset
|
103 return os.path.getmtime(self.path) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
104 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
105 @property |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
106 def was_overriden(self): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
107 return (self.flags & self.FLAG_OVERRIDEN) != 0 |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
108 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
109 @property |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
110 def num_subs(self): |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
111 return len(self.subs) |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
112 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
113 @property |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
114 def was_any_sub_baked(self): |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
115 for o in self.subs: |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
116 if o.was_baked: |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
117 return True |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
118 return False |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
119 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
120 @property |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
121 def all_assets(self): |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
122 for sub in self.subs: |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
123 yield from sub.assets |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
124 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
125 @property |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
126 def has_any_error(self): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
127 if len(self.errors) > 0: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
128 return True |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
129 for o in self.subs: |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
130 if len(o.errors) > 0: |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
131 return True |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
132 return False |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
133 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
134 def getSub(self, sub_index): |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
135 return self.subs[sub_index - 1] |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
136 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
137 def getAllErrors(self): |
400
c12ee6936b8c
bake: Return all errors from a bake record entry when asked for it.
Ludovic Chabant <ludovic@chabant.com>
parents:
359
diff
changeset
|
138 yield from self.errors |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
139 for o in self.subs: |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
140 yield from o.errors |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
141 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
142 def getAllUsedSourceNames(self): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
143 res = set() |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
144 for o in self.subs: |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
145 if o.render_info is not None: |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
146 for p, pinfo in o.render_info.items(): |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
147 res |= pinfo.used_source_names |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
148 return res |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
149 |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
150 def getAllUsedTaxonomyTerms(self): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
151 res = set() |
415
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
152 for o in self.subs: |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
153 if o.render_info is not None: |
0e9a94b7fdfa
bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents:
411
diff
changeset
|
154 for p, pinfo in o.render_info.items(): |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
155 res |= pinfo.used_taxonomy_terms |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
156 return res |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
157 |
23
923699e816d0
Don't try to get the name of a source that doesn't have one.
Ludovic Chabant <ludovic@chabant.com>
parents:
7
diff
changeset
|
158 |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
159 class TransitionalBakeRecord(TransitionalRecord): |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
160 def __init__(self, previous_path=None): |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
161 super(TransitionalBakeRecord, self).__init__(BakeRecord, |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
162 previous_path) |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
163 self.dirty_source_names = set() |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
164 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
165 def addEntry(self, entry): |
91
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
166 if (self.previous.bake_time and |
e88e330eb8dc
Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents:
85
diff
changeset
|
167 entry.path_mtime >= self.previous.bake_time): |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
168 entry.flags |= BakeRecordEntry.FLAG_SOURCE_MODIFIED |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
169 self.dirty_source_names.add(entry.source_name) |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
170 super(TransitionalBakeRecord, self).addEntry(entry) |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
171 |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
172 def getTransitionKey(self, entry): |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
173 return _get_transition_key(entry.path, entry.taxonomy_info) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
174 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
175 def getPreviousAndCurrentEntries(self, path, taxonomy_info=None): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
176 key = _get_transition_key(path, taxonomy_info) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
177 pair = self.transitions.get(key) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
178 return pair |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
179 |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
180 def getOverrideEntry(self, path, uri): |
50
2fec3ee1298f
Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents:
23
diff
changeset
|
181 for pair in self.transitions.values(): |
2fec3ee1298f
Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents:
23
diff
changeset
|
182 cur = pair[1] |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
183 if cur and cur.path != path: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
184 for o in cur.subs: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
185 if o.out_uri == uri: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
186 return cur |
50
2fec3ee1298f
Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents:
23
diff
changeset
|
187 return None |
2fec3ee1298f
Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents:
23
diff
changeset
|
188 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
189 def getPreviousEntry(self, path, taxonomy_info=None): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
190 pair = self.getPreviousAndCurrentEntries(path, taxonomy_info) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
191 if pair is not None: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
192 return pair[0] |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
193 return None |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
194 |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
195 def getCurrentEntry(self, path, taxonomy_info=None): |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
196 pair = self.getPreviousAndCurrentEntries(path, taxonomy_info) |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
197 if pair is not None: |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
198 return pair[1] |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
199 return None |
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
200 |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
201 def collapseEntry(self, prev_entry): |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
202 cur_entry = copy.deepcopy(prev_entry) |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
203 cur_entry.flags = BakeRecordEntry.FLAG_NONE |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
204 for o in cur_entry.subs: |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
205 o.flags = SubPageBakeInfo.FLAG_NONE |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
206 self.addEntry(cur_entry) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
207 |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
208 def getDeletions(self): |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
209 for prev, cur in self.transitions.values(): |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
210 if prev and not cur: |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
211 for sub in prev.subs: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
212 yield (sub.out_path, 'previous source file was removed') |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
213 elif prev and cur: |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
214 prev_out_paths = [o.out_path for o in prev.subs] |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
215 cur_out_paths = [o.out_path for o in cur.subs] |
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
216 diff = set(prev_out_paths) - set(cur_out_paths) |
120
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
217 for p in diff: |
133845647083
Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents:
96
diff
changeset
|
218 yield (p, 'source file changed outputs') |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
219 |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
220 def _onNewEntryAdded(self, entry): |
411
e7b865f8f335
bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents:
400
diff
changeset
|
221 entry.flags |= BakeRecordEntry.FLAG_NEW |
338
938be93215cb
bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents:
334
diff
changeset
|
222 |