Mercurial > piecrust2

annotate piecrust/baking/records.py @ 702:c62d83e17abf

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Some more optimizations. Write the output files in a background thread to get away from the GIL.
author Ludovic Chabant <ludovic@chabant.com>
date Sat, 16 Apr 2016 22:50:07 -0700
parents 33ab9badfd7a
children ab5c6a8ae90a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
1 import copy
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
2 import os.path
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
3 import hashlib
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import logging
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
5 from piecrust.records import Record, TransitionalRecord
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
11 def _get_transition_key(path, taxonomy_info=None):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
12 key = path
334
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
13 if taxonomy_info:
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
14 key += '+%s:%s=' % (taxonomy_info.source_name,
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
15 taxonomy_info.taxonomy_name)
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
16 if isinstance(taxonomy_info.term, tuple):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
17 key += '/'.join(taxonomy_info.term)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 else:
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
19 key += taxonomy_info.term
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
20 return hashlib.md5(key.encode('utf8')).hexdigest()
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 class BakeRecord(Record):
698
33ab9badfd7a render: Change how we store render passes info.
Ludovic Chabant <ludovic@chabant.com>
parents: 691
diff changeset
24 RECORD_VERSION = 18
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
25
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 super(BakeRecord, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 self.out_dir = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29 self.bake_time = None
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
30 self.baked_count = {}
691
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 687
diff changeset
31 self.total_baked_count = {}
217
1f4c3dae1fe8 bake: Better error handling for site baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
32 self.success = True
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
33
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
34
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
35 class SubPageBakeInfo(object):
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
36 FLAG_NONE = 0
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
37 FLAG_BAKED = 2**0
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
38 FLAG_FORCED_BY_SOURCE = 2**1
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
39 FLAG_FORCED_BY_NO_PREVIOUS = 2**2
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
40 FLAG_FORCED_BY_PREVIOUS_ERRORS = 2**3
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
41 FLAG_FORMATTING_INVALIDATED = 2**4
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
42
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
43 def __init__(self, out_uri, out_path):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
44 self.out_uri = out_uri
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
45 self.out_path = out_path
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
46 self.flags = self.FLAG_NONE
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
47 self.errors = []
698
33ab9badfd7a render: Change how we store render passes info.
Ludovic Chabant <ludovic@chabant.com>
parents: 691
diff changeset
48 self.render_info = [None, None] # Same length as RENDER_PASSES
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
49
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
50 @property
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
51 def was_clean(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
52 return (self.flags & self.FLAG_BAKED) == 0 and len(self.errors) == 0
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
53
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
54 @property
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
55 def was_baked(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
56 return (self.flags & self.FLAG_BAKED) != 0
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
57
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
58 @property
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
59 def was_baked_successfully(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
60 return self.was_baked and len(self.errors) == 0
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
61
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
62 def anyPass(self, func):
698
33ab9badfd7a render: Change how we store render passes info.
Ludovic Chabant <ludovic@chabant.com>
parents: 691
diff changeset
63 for pinfo in self.render_info:
33ab9badfd7a render: Change how we store render passes info.
Ludovic Chabant <ludovic@chabant.com>
parents: 691
diff changeset
64 if pinfo and func(pinfo):
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
65 return True
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
66 return False
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
68 def copyRenderInfo(self):
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
69 return copy.deepcopy(self.render_info)
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
70
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
71
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
72 class TaxonomyInfo(object):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
73 def __init__(self, taxonomy_name, source_name, term):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
74 self.taxonomy_name = taxonomy_name
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
75 self.source_name = source_name
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
76 self.term = term
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
77
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
78
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
79 class BakeRecordEntry(object):
334
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
80 """ An entry in the bake record.
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
81
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
82 The `taxonomy_info` attribute should be a tuple of the form:
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
83 (taxonomy name, term, source name)
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
84 """
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
85 FLAG_NONE = 0
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
86 FLAG_NEW = 2**0
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
87 FLAG_SOURCE_MODIFIED = 2**1
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
88 FLAG_OVERRIDEN = 2**2
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
89
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
90 def __init__(self, source_name, path, taxonomy_info=None):
334
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
91 self.source_name = source_name
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
92 self.path = path
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
93 self.taxonomy_info = taxonomy_info
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
94 self.flags = self.FLAG_NONE
50
2fec3ee1298f Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents: 23
diff changeset
95 self.config = None
359
2cd2b5d07129 bake: Fix crash when handling bake errors.
Ludovic Chabant <ludovic@chabant.com>
parents: 338
diff changeset
96 self.errors = []
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
97 self.subs = []
334
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
98
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
99 @property
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
100 def path_mtime(self):
b034f6f15e22 bake: Several bug taxonomy-related fixes for incorrect incremental bakes.
Ludovic Chabant <ludovic@chabant.com>
parents: 217
diff changeset
101 return os.path.getmtime(self.path)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
102
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
103 @property
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
104 def was_overriden(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
105 return (self.flags & self.FLAG_OVERRIDEN) != 0
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
106
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
107 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
108 def num_subs(self):
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
109 return len(self.subs)
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
110
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
111 @property
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
112 def was_any_sub_baked(self):
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
113 for o in self.subs:
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
114 if o.was_baked:
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
115 return True
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
116 return False
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
117
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
118 @property
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
119 def all_assets(self):
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
120 for sub in self.subs:
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
121 yield from sub.assets
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
122
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
123 @property
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
124 def has_any_error(self):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
125 if len(self.errors) > 0:
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
126 return True
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
127 for o in self.subs:
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
128 if len(o.errors) > 0:
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
129 return True
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
130 return False
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
131
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
132 def getSub(self, sub_index):
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
133 return self.subs[sub_index - 1]
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
134
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
135 def getAllErrors(self):
400
c12ee6936b8c bake: Return all errors from a bake record entry when asked for it.
Ludovic Chabant <ludovic@chabant.com>
parents: 359
diff changeset
136 yield from self.errors
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
137 for o in self.subs:
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
138 yield from o.errors
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
139
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
140 def getAllUsedSourceNames(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
141 res = set()
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
142 for o in self.subs:
698
33ab9badfd7a render: Change how we store render passes info.
Ludovic Chabant <ludovic@chabant.com>
parents: 691
diff changeset
143 for pinfo in o.render_info:
33ab9badfd7a render: Change how we store render passes info.
Ludovic Chabant <ludovic@chabant.com>
parents: 691
diff changeset
144 if pinfo:
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
145 res |= pinfo.used_source_names
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
146 return res
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
147
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
148 def getAllUsedTaxonomyTerms(self):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
149 res = set()
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 411
diff changeset
150 for o in self.subs:
698
33ab9badfd7a render: Change how we store render passes info.
Ludovic Chabant <ludovic@chabant.com>
parents: 691
diff changeset
151 for pinfo in o.render_info:
33ab9badfd7a render: Change how we store render passes info.
Ludovic Chabant <ludovic@chabant.com>
parents: 691
diff changeset
152 if pinfo:
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
153 res |= pinfo.used_taxonomy_terms
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
154 return res
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
155
23
923699e816d0 Don't try to get the name of a source that doesn't have one.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
156
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
157 class TransitionalBakeRecord(TransitionalRecord):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
158 def __init__(self, previous_path=None):
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
159 super(TransitionalBakeRecord, self).__init__(BakeRecord,
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
160 previous_path)
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
161 self.dirty_source_names = set()
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
162
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
163 def addEntry(self, entry):
91
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
164 if (self.previous.bake_time and
e88e330eb8dc Improvements to incremental baking and cache invalidating.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
165 entry.path_mtime >= self.previous.bake_time):
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
166 entry.flags |= BakeRecordEntry.FLAG_SOURCE_MODIFIED
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
167 self.dirty_source_names.add(entry.source_name)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
168 super(TransitionalBakeRecord, self).addEntry(entry)
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
169
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
170 def getTransitionKey(self, entry):
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
171 return _get_transition_key(entry.path, entry.taxonomy_info)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
172
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
173 def getPreviousAndCurrentEntries(self, path, taxonomy_info=None):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
174 key = _get_transition_key(path, taxonomy_info)
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
175 pair = self.transitions.get(key)
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
176 return pair
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
177
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
178 def getOverrideEntry(self, path, uri):
50
2fec3ee1298f Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents: 23
diff changeset
179 for pair in self.transitions.values():
2fec3ee1298f Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents: 23
diff changeset
180 cur = pair[1]
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
181 if cur and cur.path != path:
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
182 for o in cur.subs:
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
183 if o.out_uri == uri:
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
184 return cur
50
2fec3ee1298f Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents: 23
diff changeset
185 return None
2fec3ee1298f Properly override pages between realms.
Ludovic Chabant <ludovic@chabant.com>
parents: 23
diff changeset
186
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
187 def getPreviousEntry(self, path, taxonomy_info=None):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
188 pair = self.getPreviousAndCurrentEntries(path, taxonomy_info)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
189 if pair is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
190 return pair[0]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
191 return None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
192
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
193 def getCurrentEntry(self, path, taxonomy_info=None):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
194 pair = self.getPreviousAndCurrentEntries(path, taxonomy_info)
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
195 if pair is not None:
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
196 return pair[1]
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
197 return None
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
198
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
199 def collapseEntry(self, prev_entry):
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
200 cur_entry = copy.deepcopy(prev_entry)
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
201 cur_entry.flags = BakeRecordEntry.FLAG_NONE
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
202 for o in cur_entry.subs:
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
203 o.flags = SubPageBakeInfo.FLAG_NONE
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
204 self.addEntry(cur_entry)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
205
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
206 def getDeletions(self):
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
207 for prev, cur in self.transitions.values():
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
208 if prev and not cur:
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
209 for sub in prev.subs:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
210 yield (sub.out_path, 'previous source file was removed')
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
211 elif prev and cur:
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
212 prev_out_paths = [o.out_path for o in prev.subs]
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
213 cur_out_paths = [o.out_path for o in cur.subs]
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
214 diff = set(prev_out_paths) - set(cur_out_paths)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
215 for p in diff:
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 96
diff changeset
216 yield (p, 'source file changed outputs')
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
217
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
218 def _onNewEntryAdded(self, entry):
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 400
diff changeset
219 entry.flags |= BakeRecordEntry.FLAG_NEW
338
938be93215cb bake: Improve render context and bake record, fix incremental bake bugs.
Ludovic Chabant <ludovic@chabant.com>
parents: 334
diff changeset
220