Mercurial > piecrust2

annotate piecrust/processing/base.py @ 215:a47580a0955b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Better error handling for the processing pipeline. Pipeline jobs now keep track of whether they've seen any errors. This is aggregated into an overall "success" flag for the processing record. Also, jobs keep going as long as there's no critical (i.e. internal) failure happening. Errors raised by processors are also better tracked: the actual processor that failed, along with the input file, are tracks in the processing record. The `bake` command returns a failure exit code if processing saw any error.
author Ludovic Chabant <ludovic@chabant.com>
date Sat, 31 Jan 2015 17:08:02 -0800
parents e34a6826a3d4
children f82262f59600
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import re
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import time
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import shutil
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import os.path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import logging
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
6 import hashlib
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7 import threading
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
8 from queue import Queue, Empty
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 from piecrust.chefutil import format_timed
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
10 from piecrust.processing.records import (
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
11 ProcessorPipelineRecordEntry, TransitionalProcessorPipelineRecord,
191
308d5180bf81 processing: Add more information to the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
12 FLAG_PROCESSED, FLAG_OVERRIDEN, FLAG_BYPASSED_STRUCTURED_PROCESSING)
308d5180bf81 processing: Add more information to the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
13 from piecrust.processing.tree import (
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
14 ProcessingTreeBuilder, ProcessingTreeRunner,
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
15 ProcessingTreeError, ProcessorError,
191
308d5180bf81 processing: Add more information to the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
16 STATE_DIRTY,
308d5180bf81 processing: Add more information to the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
17 print_node, get_node_name_tree)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 PRIORITY_FIRST = -1
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 PRIORITY_NORMAL = 0
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 PRIORITY_LAST = 1
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27
203
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
28 split_processor_names_re = re.compile(r'[ ,]+')
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
29
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
30
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 class Processor(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 PROCESSOR_NAME = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35 self.priority = PRIORITY_NORMAL
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 self.is_bypassing_structured_processing = False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 self.is_delegating_dependency_check = True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 def initialize(self, app):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 self.app = app
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 def onPipelineStart(self, pipeline):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 pass
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 def onPipelineEnd(self, pipeline):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 pass
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47
117
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
48 def matches(self, path):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 return False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 def getDependencies(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 return None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 def getOutputFilenames(self, filename):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 return None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 def process(self, path, out_dir):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 pass
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61 class CopyFileProcessor(Processor):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62 PROCESSOR_NAME = 'copy'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 super(CopyFileProcessor, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66 self.priority = PRIORITY_LAST
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67
117
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
68 def matches(self, path):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 return True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 def getOutputFilenames(self, filename):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 return [filename]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 def process(self, path, out_dir):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 out_path = os.path.join(out_dir, os.path.basename(path))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 logger.debug("Copying: %s -> %s" % (path, out_path))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 shutil.copyfile(path, out_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 return True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81 class SimpleFileProcessor(Processor):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82 def __init__(self, extensions=None):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83 super(SimpleFileProcessor, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84 self.extensions = extensions or {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
85
117
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
86 def matches(self, path):
55
45828c4167ad Processors can match on other things than just the extension.
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
87 for ext in self.extensions:
117
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
88 if path.endswith('.' + ext):
55
45828c4167ad Processors can match on other things than just the extension.
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
89 return True
45828c4167ad Processors can match on other things than just the extension.
Ludovic Chabant <ludovic@chabant.com>
parents: 39
diff changeset
90 return False
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
91
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
92 def getOutputFilenames(self, filename):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
93 basename, ext = os.path.splitext(filename)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
94 ext = ext.lstrip('.')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
95 out_ext = self.extensions[ext]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
96 return ['%s.%s' % (basename, out_ext)]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
97
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98 def process(self, path, out_dir):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
99 _, in_name = os.path.split(path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
100 out_name = self.getOutputFilenames(in_name)[0]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
101 out_path = os.path.join(out_dir, out_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
102 return self._doProcess(path, out_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
103
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
104 def _doProcess(self, in_path, out_path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
105 raise NotImplementedError()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
106
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
107
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
108 class ProcessingContext(object):
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
109 def __init__(self, base_dir, mount_info, job_queue, record=None):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
110 self.base_dir = base_dir
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
111 self.mount_info = mount_info
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
112 self.job_queue = job_queue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
113 self.record = record
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
114
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
115
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
116 class ProcessorPipeline(object):
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
117 def __init__(self, app, out_dir, force=False):
126
e5cba2622d26 Check we don't give null values to the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
118 assert app and out_dir
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
119 self.app = app
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
120 self.out_dir = out_dir
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
121 self.force = force
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
122
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
123 tmp_dir = app.cache_dir
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
124 if not tmp_dir:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
125 import tempfile
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
126 tmp_dir = os.path.join(tempfile.gettempdir(), 'piecrust')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
127 self.tmp_dir = os.path.join(tmp_dir, 'proc')
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
128
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
129 baker_params = app.config.get('baker') or {}
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
130
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
131 assets_dirs = baker_params.get('assets_dirs', app.assets_dirs)
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
132 self.mounts = make_mount_infos(assets_dirs, self.app.root_dir)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
133
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
134 self.num_workers = baker_params.get('workers', 4)
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
135
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
136 ignores = baker_params.get('ignore', [])
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
137 ignores += [
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
138 '_cache', '_counter',
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
139 'theme_info.yml',
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
140 '.DS_Store', 'Thumbs.db',
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
141 '.git*', '.hg*', '.svn']
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
142 self.skip_patterns = make_re(ignores)
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
143 self.force_patterns = make_re(baker_params.get('force', []))
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
144
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
145 self.processors = app.plugin_loader.getProcessors()
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
146
196
154b8df04829 processing: Add Compass and Sass processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 191
diff changeset
147 def addSkipPatterns(self, patterns):
154b8df04829 processing: Add Compass and Sass processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 191
diff changeset
148 self.skip_patterns += make_re(patterns)
154b8df04829 processing: Add Compass and Sass processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 191
diff changeset
149
35
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
150 def filterProcessors(self, authorized_names):
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
151 self.processors = self.getFilteredProcessors(authorized_names)
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
152
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
153 def getFilteredProcessors(self, authorized_names):
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
154 if not authorized_names or authorized_names == 'all':
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
155 return self.processors
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
156
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
157 if isinstance(authorized_names, str):
203
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
158 authorized_names = split_processor_names_re.split(authorized_names)
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
159
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
160 procs = []
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
161 has_star = 'all' in authorized_names
203
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
162 for p in self.processors:
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
163 for name in authorized_names:
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
164 if name == p.PROCESSOR_NAME:
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
165 procs.append(p)
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
166 break
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
167 if name == ('-%s' % p.PROCESSOR_NAME):
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
168 break
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
169 else:
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
170 if has_star:
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
171 procs.append(p)
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
172 return procs
29165f2f315d processing: More powerful syntax to specify pipeline processors.
Ludovic Chabant <ludovic@chabant.com>
parents: 201
diff changeset
173
133
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
174 def run(self, src_dir_or_file=None, *,
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
175 delete=True, previous_record=None, save_record=True):
117
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
176 # Invoke pre-processors.
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
177 for proc in self.processors:
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
178 proc.onPipelineStart(self)
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
179
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
180 # Sort our processors again in case the pre-process step involved
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
181 # patching the processors with some new ones.
6827dcc9d3fb Changes to the asset processing pipeline:
Ludovic Chabant <ludovic@chabant.com>
parents: 55
diff changeset
182 self.processors.sort(key=lambda p: p.priority)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
183
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
184 # Create the pipeline record.
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
185 record = TransitionalProcessorPipelineRecord()
133
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
186 record_cache = self.app.cache.getCache('proc')
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
187 record_name = (
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
188 hashlib.md5(self.out_dir.encode('utf8')).hexdigest() +
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
189 '.record')
133
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
190 if previous_record:
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
191 record.setPrevious(previous_record)
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
192 elif not self.force and record_cache.has(record_name):
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
193 t = time.clock()
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
194 record.loadPrevious(record_cache.getCachePath(record_name))
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
195 logger.debug(format_timed(t, 'loaded previous bake record',
133
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
196 colored=False))
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
197 logger.debug("Got %d entries in process record." %
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
198 len(record.previous.entries))
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
199
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
200 # Create the workers.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
201 pool = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
202 queue = Queue()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
203 abort = threading.Event()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
204 pipeline_lock = threading.Lock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
205 for i in range(self.num_workers):
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
206 ctx = ProcessingWorkerContext(self, record,
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
207 queue, abort, pipeline_lock)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
208 worker = ProcessingWorker(i, ctx)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
209 worker.start()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
210 pool.append(worker)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
211
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
212 if src_dir_or_file is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
213 # Process only the given path.
36
485682a6de50 New site layout support.
Ludovic Chabant <ludovic@chabant.com>
parents: 35
diff changeset
214 # Find out what mount point this is in.
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
215 for name, info in self.mounts.items():
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
216 path = info['path']
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
217 if src_dir_or_file[:len(path)] == path:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
218 base_dir = path
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
219 mount_info = info
36
485682a6de50 New site layout support.
Ludovic Chabant <ludovic@chabant.com>
parents: 35
diff changeset
220 break
485682a6de50 New site layout support.
Ludovic Chabant <ludovic@chabant.com>
parents: 35
diff changeset
221 else:
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
222 known_roots = [i['path'] for i in self.mounts.values()]
36
485682a6de50 New site layout support.
Ludovic Chabant <ludovic@chabant.com>
parents: 35
diff changeset
223 raise Exception("Input path '%s' is not part of any known "
485682a6de50 New site layout support.
Ludovic Chabant <ludovic@chabant.com>
parents: 35
diff changeset
224 "mount point: %s" %
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
225 (src_dir_or_file, known_roots))
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
226
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
227 ctx = ProcessingContext(base_dir, mount_info, queue, record)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
228 logger.debug("Initiating processing pipeline on: %s" % src_dir_or_file)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
229 if os.path.isdir(src_dir_or_file):
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
230 self.processDirectory(ctx, src_dir_or_file)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
231 elif os.path.isfile(src_dir_or_file):
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
232 self.processFile(ctx, src_dir_or_file)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
233
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
234 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
235 # Process everything.
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
236 for name, info in self.mounts.items():
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
237 path = info['path']
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
238 ctx = ProcessingContext(path, info, queue, record)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
239 logger.debug("Initiating processing pipeline on: %s" % path)
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
240 self.processDirectory(ctx, path)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
241
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
242 # Wait on all workers.
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
243 record.current.success = True
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
244 for w in pool:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
245 w.join()
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
246 record.current.success &= w.success
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
247 if abort.is_set():
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
248 raise Exception("Worker pool was aborted.")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
249
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
250 # Handle deletions.
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
251 if delete:
129
3080b6d02f40 Add ability for the processing pipeline to only process new assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 126
diff changeset
252 for path, reason in record.getDeletions():
3080b6d02f40 Add ability for the processing pipeline to only process new assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 126
diff changeset
253 logger.debug("Removing '%s': %s" % (path, reason))
207
c5330cb35794 processing: Don't fail if an asset we want to remove has already been removed.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
254 try:
c5330cb35794 processing: Don't fail if an asset we want to remove has already been removed.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
255 os.remove(path)
c5330cb35794 processing: Don't fail if an asset we want to remove has already been removed.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
256 except FileNotFoundError:
c5330cb35794 processing: Don't fail if an asset we want to remove has already been removed.
Ludovic Chabant <ludovic@chabant.com>
parents: 205
diff changeset
257 pass
129
3080b6d02f40 Add ability for the processing pipeline to only process new assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 126
diff changeset
258 logger.info('[delete] %s' % path)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
259
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
260 # Invoke post-processors.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
261 for proc in self.processors:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
262 proc.onPipelineEnd(self)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
263
133
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
264 # Finalize the process record.
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
265 record.current.process_time = time.time()
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
266 record.current.out_dir = self.out_dir
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
267 record.collapseRecords()
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
268
133
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
269 # Save the process record.
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
270 if save_record:
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
271 t = time.clock()
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
272 record.saveCurrent(record_cache.getCachePath(record_name))
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
273 logger.debug(format_timed(t, 'saved bake record', colored=False))
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
274
9e4c2e68a129 Optimize server for files that already exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 129
diff changeset
275 return record.detach()
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
276
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
277 def processDirectory(self, ctx, start_dir):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
278 for dirpath, dirnames, filenames in os.walk(start_dir):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
279 rel_dirpath = os.path.relpath(dirpath, start_dir)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
280 dirnames[:] = [d for d in dirnames
35
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
281 if not re_matchany(d, self.skip_patterns, rel_dirpath)]
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
282
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
283 for filename in filenames:
35
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
284 if re_matchany(filename, self.skip_patterns, rel_dirpath):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
285 continue
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
286 self.processFile(ctx, os.path.join(dirpath, filename))
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
287
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
288 def processFile(self, ctx, path):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
289 logger.debug("Queuing: %s" % path)
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
290 job = ProcessingWorkerJob(ctx.base_dir, ctx.mount_info, path)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
291 ctx.job_queue.put_nowait(job)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
292
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
293
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
294 class ProcessingWorkerContext(object):
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
295 def __init__(self, pipeline, record,
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
296 work_queue, abort_event, pipeline_lock):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
297 self.pipeline = pipeline
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
298 self.record = record
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
299 self.work_queue = work_queue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
300 self.abort_event = abort_event
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
301 self.pipeline_lock = pipeline_lock
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
302
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
303
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
304 class ProcessingWorkerJob(object):
213
e34a6826a3d4 internal: Remove the (unused) `new_only` flag for pipeline processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 208
diff changeset
305 def __init__(self, base_dir, mount_info, path):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
306 self.base_dir = base_dir
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
307 self.mount_info = mount_info
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
308 self.path = path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
309
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
310
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
311 class ProcessingWorker(threading.Thread):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
312 def __init__(self, wid, ctx):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
313 super(ProcessingWorker, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
314 self.wid = wid
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
315 self.ctx = ctx
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
316 self.success = True
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
317
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
318 def run(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
319 while(not self.ctx.abort_event.is_set()):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
320 try:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
321 job = self.ctx.work_queue.get(True, 0.1)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
322 except Empty:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
323 logger.debug("[%d] No more work... shutting down." % self.wid)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
324 break
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
325
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
326 try:
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
327 success = self._unsafeRun(job)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
328 logger.debug("[%d] Done with file." % self.wid)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
329 self.ctx.work_queue.task_done()
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
330 self.success &= success
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
331 except Exception as ex:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
332 self.ctx.abort_event.set()
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
333 self.success = False
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
334 logger.error("[%d] Critical error, aborting." % self.wid)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
335 logger.exception(ex)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
336 break
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
337
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
338 def _unsafeRun(self, job):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
339 start_time = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
340 pipeline = self.ctx.pipeline
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
341 record = self.ctx.record
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
342
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
343 rel_path = os.path.relpath(job.path, job.base_dir)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
344 previous_entry = record.getPreviousEntry(rel_path)
129
3080b6d02f40 Add ability for the processing pipeline to only process new assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 126
diff changeset
345
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
346 record_entry = ProcessorPipelineRecordEntry(job.base_dir, rel_path)
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
347 record.addEntry(record_entry)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
348
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
349 # Figure out if a previously processed file is overriding this one.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
350 # This can happen if a theme file (processed via a mount point)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
351 # is overridden in the user's website.
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
352 if record.current.hasOverrideEntry(rel_path):
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
353 record_entry.flags |= FLAG_OVERRIDEN
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
354 logger.info(format_timed(start_time,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
355 '%s [not baked, overridden]' % rel_path))
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
356 return True
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
357
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
358 processors = pipeline.getFilteredProcessors(
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
359 job.mount_info['processors'])
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
360 try:
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
361 builder = ProcessingTreeBuilder(processors)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
362 tree_root = builder.build(rel_path)
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
363 except ProcessingTreeError as ex:
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
364 msg = str(ex)
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
365 logger.error("Error processing %s: %s" % (rel_path, msg))
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
366 while ex:
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
367 record_entry.errors.append(str(ex))
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
368 ex = ex.__cause__
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
369 return False
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
370
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
371 print_node(tree_root, recursive=True)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
372 leaves = tree_root.getLeaves()
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
373 record_entry.rel_outputs = [l.path for l in leaves]
191
308d5180bf81 processing: Add more information to the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
374 record_entry.proc_tree = get_node_name_tree(tree_root)
308d5180bf81 processing: Add more information to the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
375 if tree_root.getProcessor().is_bypassing_structured_processing:
308d5180bf81 processing: Add more information to the pipeline record.
Ludovic Chabant <ludovic@chabant.com>
parents: 133
diff changeset
376 record_entry.flags |= FLAG_BYPASSED_STRUCTURED_PROCESSING
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
377
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
378 force = (pipeline.force or previous_entry is None or
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
379 not previous_entry.was_processed_successfully)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
380 if not force:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
381 force = re_matchany(rel_path, pipeline.force_patterns)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
382
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
383 if force:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
384 tree_root.setState(STATE_DIRTY, True)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
385
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
386 try:
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
387 runner = ProcessingTreeRunner(
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
388 job.base_dir, pipeline.tmp_dir,
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
389 pipeline.out_dir, self.ctx.pipeline_lock)
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
390 if runner.processSubTree(tree_root):
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
391 record_entry.flags |= FLAG_PROCESSED
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
392 logger.info(format_timed(
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
393 start_time, "[%d] %s" % (self.wid, rel_path)))
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
394 return True
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 117
diff changeset
395 except ProcessingTreeError as ex:
215
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
396 msg = str(ex)
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
397 if isinstance(ex, ProcessorError):
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
398 msg = str(ex.__cause__)
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
399 logger.error("Error processing %s: %s" % (rel_path, msg))
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
400 while ex:
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
401 record_entry.errors.append(str(ex))
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
402 ex = ex.__cause__
a47580a0955b bake: Better error handling for the processing pipeline.
Ludovic Chabant <ludovic@chabant.com>
parents: 213
diff changeset
403 return False
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
404
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
405
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
406 def make_mount_infos(mounts, root_dir):
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
407 if isinstance(mounts, list):
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
408 mounts = {m: {} for m in mounts}
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
409
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
410 for name, info in mounts.items():
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
411 if not isinstance(info, dict):
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
412 raise Exception("Asset directory info for '%s' is not a "
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
413 "dictionary." % name)
205
e725af1d48fb bake: Changes in how assets directories are configured.
Ludovic Chabant <ludovic@chabant.com>
parents: 203
diff changeset
414 info.setdefault('processors', 'all -uglifyjs -cleancss')
208
989d0abd7c17 processing: Use the correct full path for mounts.
Ludovic Chabant <ludovic@chabant.com>
parents: 207
diff changeset
415 info['path'] = os.path.join(root_dir, name)
201
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
416
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
417 return mounts
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
418
0c9de41689bb processing: Add ability to specify processors per mount.
Ludovic Chabant <ludovic@chabant.com>
parents: 196
diff changeset
419
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
420 def make_re(patterns):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
421 re_patterns = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
422 for pat in patterns:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
423 if pat[0] == '/' and pat[-1] == '/' and len(pat) > 2:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
424 re_patterns.append(pat[1:-1])
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
425 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
426 escaped_pat = (re.escape(pat)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
427 .replace(r'\*', r'[^/\\]*')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
428 .replace(r'\?', r'[^/\\]'))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
429 re_patterns.append(escaped_pat)
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
430 return [re.compile(p) for p in re_patterns]
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
431
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
432
35
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
433 def re_matchany(filename, patterns, dirname=None):
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
434 if dirname and dirname != '.':
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
435 filename = os.path.join(dirname, filename)
e4c345dcf33c More unit tests, fix a bug with the skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 22
diff changeset
436
11
617191dec18e Fixes for Windows, make `findPagePath` return a ref path.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
437 # skip patterns use a forward slash regardless of the platform.
617191dec18e Fixes for Windows, make `findPagePath` return a ref path.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
438 filename = filename.replace('\\', '/')
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
439 for pattern in patterns:
22
df790a827d38 Correctly match skip patterns.
Ludovic Chabant <ludovic@chabant.com>
parents: 11
diff changeset
440 if pattern.search(filename):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
441 return True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
442 return False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
443