Mercurial > piecrust2

annotate piecrust/processing/base.py @ 230:016d42c23ba9

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
internal: Make the simple page source use `slug` everywhere. Before, it would use `slug` in some places, and `path` in other places, which would be quite confusing.
author Ludovic Chabant <ludovic@chabant.com>
date Wed, 11 Feb 2015 08:29:09 -0800
parents f82262f59600
children c2ca72fb7f0b
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import re
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import time
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import shutil
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import os.path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import logging
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
6 import hashlib
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7 import threading
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
8 from queue import Queue, Empty
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 from piecrust.chefutil import format_timed
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
10 from piecrust.processing.records import (
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
11 ProcessorPipelineRecordEntry, TransitionalProcessorPipelineRecord,
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
12 FLAG_PREPARED, FLAG_PROCESSED, FLAG_OVERRIDEN,
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
13 FLAG_BYPASSED_STRUCTURED_PROCESSING)
191
308d5180bf81 processing: Add more information to the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
14 from piecrust.processing.tree import (
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
15 ProcessingTreeBuilder, ProcessingTreeRunner,
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
16 ProcessingTreeError, ProcessorError,
191
308d5180bf81 processing: Add more information to the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
17 STATE_DIRTY,
308d5180bf81 processing: Add more information to the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
18 print_node, get_node_name_tree)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
24 re_ansicolors = re.compile('\033\\[\d+m')
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
25
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
26
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 PRIORITY_FIRST = -1
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 PRIORITY_NORMAL = 0
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29 PRIORITY_LAST = 1
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31
203
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
32 split_processor_names_re = re.compile(r'[ ,]+')
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
33
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
34
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35 class Processor(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 PROCESSOR_NAME = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 self.priority = PRIORITY_NORMAL
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 self.is_bypassing_structured_processing = False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 self.is_delegating_dependency_check = True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 def initialize(self, app):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 self.app = app
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 def onPipelineStart(self, pipeline):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 pass
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 def onPipelineEnd(self, pipeline):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 pass
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51
117
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
52 def matches(self, path):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 return False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 def getDependencies(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56 return None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 def getOutputFilenames(self, filename):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 return None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61 def process(self, path, out_dir):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62 pass
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 class CopyFileProcessor(Processor):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66 PROCESSOR_NAME = 'copy'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 super(CopyFileProcessor, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70 self.priority = PRIORITY_LAST
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71
117
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
72 def matches(self, path):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 return True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 def getOutputFilenames(self, filename):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 return [filename]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 def process(self, path, out_dir):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 out_path = os.path.join(out_dir, os.path.basename(path))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 logger.debug("Copying: %s -> %s" % (path, out_path))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81 shutil.copyfile(path, out_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82 return True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
85 class SimpleFileProcessor(Processor):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
86 def __init__(self, extensions=None):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
87 super(SimpleFileProcessor, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
88 self.extensions = extensions or {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
89
117
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
90 def matches(self, path):
55
45828c4167ad Processors can match on other things than just the extension.
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
91 for ext in self.extensions:
117
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
92 if path.endswith('.' + ext):
55
45828c4167ad Processors can match on other things than just the extension.
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
93 return True
45828c4167ad Processors can match on other things than just the extension.
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
94 return False
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
95
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
96 def getOutputFilenames(self, filename):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
97 basename, ext = os.path.splitext(filename)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98 ext = ext.lstrip('.')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
99 out_ext = self.extensions[ext]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
100 return ['%s.%s' % (basename, out_ext)]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
101
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
102 def process(self, path, out_dir):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
103 _, in_name = os.path.split(path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
104 out_name = self.getOutputFilenames(in_name)[0]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
105 out_path = os.path.join(out_dir, out_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
106 return self._doProcess(path, out_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
107
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
108 def _doProcess(self, in_path, out_path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
109 raise NotImplementedError()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
110
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
111
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
112 class ExternalProcessException(Exception):
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
113 def __init__(self, stderr_data):
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
114 self.stderr_data = stderr_data
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
115
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
116 def __str__(self):
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
117 return self.stderr_data
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
118
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
119
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
120 class ProcessingContext(object):
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
121 def __init__(self, base_dir, mount_info, job_queue, record=None):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
122 self.base_dir = base_dir
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
123 self.mount_info = mount_info
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
124 self.job_queue = job_queue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
125 self.record = record
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
126
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
127
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
128 class ProcessorPipeline(object):
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
129 def __init__(self, app, out_dir, force=False):
126
e5cba2622d26 Check we don't give null values to the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
130 assert app and out_dir
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
131 self.app = app
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
132 self.out_dir = out_dir
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
133 self.force = force
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
134
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
135 tmp_dir = app.cache_dir
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
136 if not tmp_dir:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
137 import tempfile
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
138 tmp_dir = os.path.join(tempfile.gettempdir(), 'piecrust')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
139 self.tmp_dir = os.path.join(tmp_dir, 'proc')
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
140
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
141 baker_params = app.config.get('baker') or {}
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
142
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
143 assets_dirs = baker_params.get('assets_dirs', app.assets_dirs)
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
144 self.mounts = make_mount_infos(assets_dirs, self.app.root_dir)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
145
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
146 self.num_workers = baker_params.get('workers', 4)
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
147
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
148 ignores = baker_params.get('ignore', [])
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
149 ignores += [
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
150 '_cache', '_counter',
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
151 'theme_info.yml',
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
152 '.DS_Store', 'Thumbs.db',
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
153 '.git*', '.hg*', '.svn']
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
154 self.skip_patterns = make_re(ignores)
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
155 self.force_patterns = make_re(baker_params.get('force', []))
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
156
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
157 self.processors = app.plugin_loader.getProcessors()
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
158
196
154b8df04829 processing: Add Compass and Sass processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 191
diff changeset
159 def addSkipPatterns(self, patterns):
154b8df04829 processing: Add Compass and Sass processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 191
diff changeset
160 self.skip_patterns += make_re(patterns)
154b8df04829 processing: Add Compass and Sass processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 191
diff changeset
161
35
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
162 def filterProcessors(self, authorized_names):
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
163 self.processors = self.getFilteredProcessors(authorized_names)
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
164
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
165 def getFilteredProcessors(self, authorized_names):
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
166 if not authorized_names or authorized_names == 'all':
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
167 return self.processors
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
168
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
169 if isinstance(authorized_names, str):
203
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
170 authorized_names = split_processor_names_re.split(authorized_names)
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
171
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
172 procs = []
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
173 has_star = 'all' in authorized_names
203
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
174 for p in self.processors:
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
175 for name in authorized_names:
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
176 if name == p.PROCESSOR_NAME:
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
177 procs.append(p)
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
178 break
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
179 if name == ('-%s' % p.PROCESSOR_NAME):
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
180 break
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
181 else:
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
182 if has_star:
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
183 procs.append(p)
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
184 return procs
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
185
133
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
186 def run(self, src_dir_or_file=None, *,
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
187 delete=True, previous_record=None, save_record=True):
117
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
188 # Invoke pre-processors.
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
189 for proc in self.processors:
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
190 proc.onPipelineStart(self)
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
191
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
192 # Sort our processors again in case the pre-process step involved
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
193 # patching the processors with some new ones.
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
194 self.processors.sort(key=lambda p: p.priority)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
195
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
196 # Create the pipeline record.
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
197 record = TransitionalProcessorPipelineRecord()
133
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
198 record_cache = self.app.cache.getCache('proc')
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
199 record_name = (
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
200 hashlib.md5(self.out_dir.encode('utf8')).hexdigest() +
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
201 '.record')
133
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
202 if previous_record:
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
203 record.setPrevious(previous_record)
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
204 elif not self.force and record_cache.has(record_name):
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
205 t = time.clock()
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
206 record.loadPrevious(record_cache.getCachePath(record_name))
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
207 logger.debug(format_timed(t, 'loaded previous bake record',
133
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
208 colored=False))
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
209 logger.debug("Got %d entries in process record." %
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
210 len(record.previous.entries))
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
211
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
212 # Create the workers.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
213 pool = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
214 queue = Queue()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
215 abort = threading.Event()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
216 pipeline_lock = threading.Lock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
217 for i in range(self.num_workers):
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
218 ctx = ProcessingWorkerContext(self, record,
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
219 queue, abort, pipeline_lock)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
220 worker = ProcessingWorker(i, ctx)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
221 worker.start()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
222 pool.append(worker)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
223
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
224 if src_dir_or_file is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
225 # Process only the given path.
36
485682a6de50 New site layout support.
Ludovic Chabant <ludovic@chabant.com>
parents: 35
diff changeset
226 # Find out what mount point this is in.
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
227 for name, info in self.mounts.items():
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
228 path = info['path']
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
229 if src_dir_or_file[:len(path)] == path:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
230 base_dir = path
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
231 mount_info = info
36
485682a6de50 New site layout support.
Ludovic Chabant <ludovic@chabant.com>
parents: 35
diff changeset
232 break
485682a6de50 New site layout support.
Ludovic Chabant <ludovic@chabant.com>
parents: 35
diff changeset
233 else:
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
234 known_roots = [i['path'] for i in self.mounts.values()]
36
485682a6de50 New site layout support.
Ludovic Chabant <ludovic@chabant.com>
parents: 35
diff changeset
235 raise Exception("Input path '%s' is not part of any known "
485682a6de50 New site layout support.
Ludovic Chabant <ludovic@chabant.com>
parents: 35
diff changeset
236 "mount point: %s" %
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
237 (src_dir_or_file, known_roots))
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
238
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
239 ctx = ProcessingContext(base_dir, mount_info, queue, record)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
240 logger.debug("Initiating processing pipeline on: %s" % src_dir_or_file)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
241 if os.path.isdir(src_dir_or_file):
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
242 self.processDirectory(ctx, src_dir_or_file)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
243 elif os.path.isfile(src_dir_or_file):
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
244 self.processFile(ctx, src_dir_or_file)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
245
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
246 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
247 # Process everything.
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
248 for name, info in self.mounts.items():
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
249 path = info['path']
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
250 ctx = ProcessingContext(path, info, queue, record)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
251 logger.debug("Initiating processing pipeline on: %s" % path)
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
252 self.processDirectory(ctx, path)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
253
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
254 # Wait on all workers.
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
255 record.current.success = True
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
256 for w in pool:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
257 w.join()
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
258 record.current.success &= w.success
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
259 if abort.is_set():
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
260 raise Exception("Worker pool was aborted.")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
261
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
262 # Handle deletions.
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
263 if delete:
129
3080b6d02f40 Add ability for the processing pipeline to only process new assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 126
diff changeset
264 for path, reason in record.getDeletions():
3080b6d02f40 Add ability for the processing pipeline to only process new assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 126
diff changeset
265 logger.debug("Removing '%s': %s" % (path, reason))
207
c5330cb35794 processing: Don't fail if an asset we want to remove has already been removed.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
266 try:
c5330cb35794 processing: Don't fail if an asset we want to remove has already been removed.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
267 os.remove(path)
c5330cb35794 processing: Don't fail if an asset we want to remove has already been removed.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
268 except FileNotFoundError:
c5330cb35794 processing: Don't fail if an asset we want to remove has already been removed.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
269 pass
129
3080b6d02f40 Add ability for the processing pipeline to only process new assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 126
diff changeset
270 logger.info('[delete] %s' % path)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
271
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
272 # Invoke post-processors.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
273 for proc in self.processors:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
274 proc.onPipelineEnd(self)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
275
133
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
276 # Finalize the process record.
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
277 record.current.process_time = time.time()
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
278 record.current.out_dir = self.out_dir
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
279 record.collapseRecords()
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
280
133
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
281 # Save the process record.
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
282 if save_record:
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
283 t = time.clock()
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
284 record.saveCurrent(record_cache.getCachePath(record_name))
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
285 logger.debug(format_timed(t, 'saved bake record', colored=False))
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
286
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
287 return record.detach()
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
288
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
289 def processDirectory(self, ctx, start_dir):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
290 for dirpath, dirnames, filenames in os.walk(start_dir):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
291 rel_dirpath = os.path.relpath(dirpath, start_dir)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
292 dirnames[:] = [d for d in dirnames
35
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
293 if not re_matchany(d, self.skip_patterns, rel_dirpath)]
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
294
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
295 for filename in filenames:
35
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
296 if re_matchany(filename, self.skip_patterns, rel_dirpath):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
297 continue
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
298 self.processFile(ctx, os.path.join(dirpath, filename))
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
299
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
300 def processFile(self, ctx, path):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
301 logger.debug("Queuing: %s" % path)
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
302 job = ProcessingWorkerJob(ctx.base_dir, ctx.mount_info, path)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
303 ctx.job_queue.put_nowait(job)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
304
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
305
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
306 class ProcessingWorkerContext(object):
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
307 def __init__(self, pipeline, record,
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
308 work_queue, abort_event, pipeline_lock):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
309 self.pipeline = pipeline
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
310 self.record = record
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
311 self.work_queue = work_queue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
312 self.abort_event = abort_event
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
313 self.pipeline_lock = pipeline_lock
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
314
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
315
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
316 class ProcessingWorkerJob(object):
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
317 def __init__(self, base_dir, mount_info, path):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
318 self.base_dir = base_dir
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
319 self.mount_info = mount_info
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
320 self.path = path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
321
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
322
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
323 class ProcessingWorker(threading.Thread):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
324 def __init__(self, wid, ctx):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
325 super(ProcessingWorker, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
326 self.wid = wid
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
327 self.ctx = ctx
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
328 self.success = True
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
329
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
330 def run(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
331 while(not self.ctx.abort_event.is_set()):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
332 try:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
333 job = self.ctx.work_queue.get(True, 0.1)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
334 except Empty:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
335 logger.debug("[%d] No more work... shutting down." % self.wid)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
336 break
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
337
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
338 try:
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
339 success = self._unsafeRun(job)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
340 logger.debug("[%d] Done with file." % self.wid)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
341 self.ctx.work_queue.task_done()
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
342 self.success &= success
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
343 except Exception as ex:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
344 self.ctx.abort_event.set()
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
345 self.success = False
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
346 logger.error("[%d] Critical error, aborting." % self.wid)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
347 logger.exception(ex)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
348 break
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
349
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
350 def _unsafeRun(self, job):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
351 start_time = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
352 pipeline = self.ctx.pipeline
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
353 record = self.ctx.record
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
354
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
355 rel_path = os.path.relpath(job.path, job.base_dir)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
356 previous_entry = record.getPreviousEntry(rel_path)
129
3080b6d02f40 Add ability for the processing pipeline to only process new assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 126
diff changeset
357
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
358 record_entry = ProcessorPipelineRecordEntry(job.base_dir, rel_path)
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
359 record.addEntry(record_entry)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
360
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
361 # Figure out if a previously processed file is overriding this one.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
362 # This can happen if a theme file (processed via a mount point)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
363 # is overridden in the user's website.
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
364 if record.current.hasOverrideEntry(rel_path):
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
365 record_entry.flags |= FLAG_OVERRIDEN
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
366 logger.info(format_timed(start_time,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
367 '%s [not baked, overridden]' % rel_path))
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
368 return True
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
369
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
370 processors = pipeline.getFilteredProcessors(
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
371 job.mount_info['processors'])
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
372 try:
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
373 builder = ProcessingTreeBuilder(processors)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
374 tree_root = builder.build(rel_path)
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
375 record_entry.flags |= FLAG_PREPARED
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
376 except ProcessingTreeError as ex:
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
377 msg = str(ex)
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
378 logger.error("Error preparing %s:\n%s" % (rel_path, msg))
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
379 while ex:
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
380 record_entry.errors.append(str(ex))
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
381 ex = ex.__cause__
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
382 return False
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
383
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
384 print_node(tree_root, recursive=True)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
385 leaves = tree_root.getLeaves()
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
386 record_entry.rel_outputs = [l.path for l in leaves]
191
308d5180bf81 processing: Add more information to the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
387 record_entry.proc_tree = get_node_name_tree(tree_root)
308d5180bf81 processing: Add more information to the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
388 if tree_root.getProcessor().is_bypassing_structured_processing:
308d5180bf81 processing: Add more information to the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
389 record_entry.flags |= FLAG_BYPASSED_STRUCTURED_PROCESSING
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
390
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
391 force = (pipeline.force or previous_entry is None or
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
392 not previous_entry.was_processed_successfully)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
393 if not force:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
394 force = re_matchany(rel_path, pipeline.force_patterns)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
395
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
396 if force:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
397 tree_root.setState(STATE_DIRTY, True)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
398
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
399 try:
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
400 runner = ProcessingTreeRunner(
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
401 job.base_dir, pipeline.tmp_dir,
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
402 pipeline.out_dir, self.ctx.pipeline_lock)
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
403 if runner.processSubTree(tree_root):
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
404 record_entry.flags |= FLAG_PROCESSED
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
405 logger.info(format_timed(
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
406 start_time, "[%d] %s" % (self.wid, rel_path)))
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
407 return True
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
408 except ProcessingTreeError as ex:
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
409 msg = str(ex)
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
410 if isinstance(ex, ProcessorError):
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
411 msg = str(ex.__cause__)
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
412 logger.error("Error processing %s:\n%s" % (rel_path, msg))
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
413 while ex:
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
414 msg = re_ansicolors.sub('', str(ex))
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 215
diff changeset
415 record_entry.errors.append(msg)
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
416 ex = ex.__cause__
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
417 return False
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
418
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
419
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
420 def make_mount_infos(mounts, root_dir):
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
421 if isinstance(mounts, list):
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
422 mounts = {m: {} for m in mounts}
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
423
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
424 for name, info in mounts.items():
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
425 if not isinstance(info, dict):
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
426 raise Exception("Asset directory info for '%s' is not a "
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
427 "dictionary." % name)
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
428 info.setdefault('processors', 'all -uglifyjs -cleancss')
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
429 info['path'] = os.path.join(root_dir, name)
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
430
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
431 return mounts
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
432
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
433
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
434 def make_re(patterns):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
435 re_patterns = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
436 for pat in patterns:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
437 if pat[0] == '/' and pat[-1] == '/' and len(pat) > 2:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
438 re_patterns.append(pat[1:-1])
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
439 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
440 escaped_pat = (re.escape(pat)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
441 .replace(r'\*', r'[^/\\]*')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
442 .replace(r'\?', r'[^/\\]'))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
443 re_patterns.append(escaped_pat)
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
444 return [re.compile(p) for p in re_patterns]
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
445
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
446
35
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
447 def re_matchany(filename, patterns, dirname=None):
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
448 if dirname and dirname != '.':
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
449 filename = os.path.join(dirname, filename)
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
450
11
617191dec18e Fixes for Windows, make `findPagePath` return a ref path.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
451 # skip patterns use a forward slash regardless of the platform.
617191dec18e Fixes for Windows, make `findPagePath` return a ref path.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
452 filename = filename.replace('\\', '/')
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
453 for pattern in patterns:
22
df790a827d38 Correctly match skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 11
diff changeset
454 if pattern.search(filename):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
455 return True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
456 return False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
457