Mercurial > piecrust2

annotate piecrust/processing/compressors.py @ 215:a47580a0955b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Better error handling for the processing pipeline. Pipeline jobs now keep track of whether they've seen any errors. This is aggregated into an overall "success" flag for the processing record. Also, jobs keep going as long as there's no critical (i.e. internal) failure happening. Errors raised by processors are also better tracked: the actual processor that failed, along with the input file, are tracks in the processing record. The `bake` command returns a failure exit code if processing saw any error.
author Ludovic Chabant <ludovic@chabant.com>
date Sat, 31 Jan 2015 17:08:02 -0800
parents cba781477bd0
children d1490028e211
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import logging
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import platform
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import subprocess
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 from piecrust.processing.base import SimpleFileProcessor
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 logger = logging.getLogger(__name__)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12 class CleanCssProcessor(SimpleFileProcessor):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13 PROCESSOR_NAME = 'cleancss'
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 def __init__(self):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16 super(CleanCssProcessor, self).__init__({'css': 'css'})
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 self._conf = None
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19 def _doProcess(self, in_path, out_path):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 self._ensureInitialized()
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22 args = [self._conf['bin'], '-o', out_path]
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 args += self._conf['options']
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 args.append(in_path)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 logger.debug("Cleaning CSS file: %s" % args)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 # On Windows, we need to run the process in a shell environment
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 # otherwise it looks like `PATH` isn't taken into account.
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29 shell = (platform.system() == 'Windows')
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 try:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 retcode = subprocess.call(args, shell=shell)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 except FileNotFoundError as ex:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 logger.error("Tried running CleanCSS processor with command: %s" %
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 args)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35 raise Exception("Error running CleanCSS processor. "
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 "Did you install it?") from ex
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 if retcode != 0:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 raise Exception("Error occured in CleanCSS. Please check "
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 "log messages above for more information.")
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 return True
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 def _ensureInitialized(self):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 if self._conf is not None:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 return
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 self._conf = self.app.config.get('cleancss') or {}
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 self._conf.setdefault('bin', 'cleancss')
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 self._conf.setdefault('options', ['--skip-rebase'])
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 if not isinstance(self._conf['options'], list):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 raise Exception("The `cleancss/options` configuration setting "
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 "must be an array of arguments.")
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 class UglifyJSProcessor(SimpleFileProcessor):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 PROCESSOR_NAME = 'uglifyjs'
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 def __init__(self):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 super(UglifyJSProcessor, self).__init__({'js': 'js'})
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 self._conf = None
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61 def _doProcess(self, in_path, out_path):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62 self._ensureInitialized()
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 args = [self._conf['bin'], in_path, '-o', out_path]
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 args += self._conf['options']
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66 logger.debug("Uglifying JS file: %s" % args)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68 # On Windows, we need to run the process in a shell environment
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 # otherwise it looks like `PATH` isn't taken into account.
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70 shell = (platform.system() == 'Windows')
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 try:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 retcode = subprocess.call(args, shell=shell)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 except FileNotFoundError as ex:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 logger.error("Tried running UglifyJS processor with command: %s" %
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 args)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 raise Exception("Error running UglifyJS processor. "
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 "Did you install it?") from ex
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 if retcode != 0:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 raise Exception("Error occured in UglifyJS. Please check "
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 "log messages above for more information.")
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81 return True
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83 def _ensureInitialized(self):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84 if self._conf is not None:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
85 return
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
86
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
87 self._conf = self.app.config.get('uglifyjs') or {}
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
88 self._conf.setdefault('bin', 'uglifyjs')
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
89 self._conf.setdefault('options', ['--compress'])
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
90 if not isinstance(self._conf['options'], list):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
91 raise Exception("The `uglify/options` configuration setting "
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
92 "must be an array of arguments.")
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
93