Mercurial > piecrust2

annotate piecrust/processing/util.py @ 215:a47580a0955b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Better error handling for the processing pipeline. Pipeline jobs now keep track of whether they've seen any errors. This is aggregated into an overall "success" flag for the processing record. Also, jobs keep going as long as there's no critical (i.e. internal) failure happening. Errors raised by processors are also better tracked: the actual processor that failed, along with the input file, are tracks in the processing record. The `bake` command returns a failure exit code if processing saw any error.
author Ludovic Chabant <ludovic@chabant.com>
date Sat, 31 Jan 2015 17:08:02 -0800
parents cba781477bd0
children 46025a1b5434
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os.path
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import time
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import logging
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import yaml
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 from piecrust.processing.base import Processor
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 logger = logging.getLogger(__name__)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11 class _ConcatInfo(object):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12 timestamp = 0
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13 files = None
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 delim = "\n"
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 class ConcatProcessor(Processor):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 PROCESSOR_NAME = 'concat'
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 def __init__(self):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 super(ConcatProcessor, self).__init__()
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22 self._cache = {}
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 def matches(self, path):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 return path.endswith('.concat')
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 def getDependencies(self, path):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 info = self._load(path)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29 return info.files
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 def getOutputFilenames(self, filename):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 return [filename[:-7]]
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 def process(self, path, out_dir):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35 dirname, filename = os.path.split(path)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 out_path = os.path.join(out_dir, filename[:-7])
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 info = self._load(path)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 if not info.files:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 raise Exception("No files specified in: %s" %
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 os.path.relpath(path, self.app.root_dir))
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 logger.debug("Concatenating %d files to: %s" %
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 (len(info.files), out_path))
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 encoded_delim = info.delim.encode('utf8')
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 with open(out_path, 'wb') as ofp:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 for p in info.files:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 with open(p, 'rb') as ifp:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 ofp.write(ifp.read())
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 if info.delim:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 ofp.write(encoded_delim)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 return True
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 def _load(self, path):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 cur_time = time.time()
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 info = self._cache.get(path)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56 if (info is not None and
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 (cur_time - info.timestamp <= 1 or
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 os.path.getmtime(path) < info.timestamp)):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 return info
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61 if info is None:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62 info = _ConcatInfo()
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63 self._cache[path] = info
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 with open(path, 'r') as fp:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66 config = yaml.load(fp)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68 info.files = config.get('files', [])
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 info.delim = config.get('delim', "\n")
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70 info.timestamp = cur_time
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 path_mode = config.get('path_mode', 'relative')
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 if path_mode == 'relative':
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 dirname, _ = os.path.split(path)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 info.files = [os.path.join(dirname, f) for f in info.files]
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 elif path_mode == 'absolute':
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 info.files = [os.path.join(self.app.root_dir, f)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 for f in info.files]
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 else:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 raise Exception("Unknown path mode: %s" % path_mode)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82 return info
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83