Mercurial > piecrust2

annotate piecrust/processing/compressors.py @ 1188:a7c43131d871

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 15 Jun 2021 22:36:23 -0700
parents a3dec0fbd9ce
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import logging
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import platform
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import subprocess
315
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
6 from piecrust.processing.base import Processor, SimpleFileProcessor
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 logger = logging.getLogger(__name__)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
315
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
12 class CleanCssProcessor(Processor):
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13 PROCESSOR_NAME = 'cleancss'
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 def __init__(self):
315
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
16 super(CleanCssProcessor, self).__init__()
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 self._conf = None
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
315
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
19 def matches(self, path):
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 339
diff changeset
20 return path.endswith('.css') and not path.endswith('.min.css')
315
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
21
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
22 def getOutputFilenames(self, filename):
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 self._ensureInitialized()
315
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
24 basename, _ = os.path.splitext(filename)
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
25 return ['%s%s' % (basename, self._conf['out_ext'])]
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
26
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
27 def process(self, path, out_dir):
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
28 self._ensureInitialized()
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
29
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
30 _, in_name = os.path.split(path)
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
31 out_name = self.getOutputFilenames(in_name)[0]
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
32 out_path = os.path.join(out_dir, out_name)
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 args = [self._conf['bin'], '-o', out_path]
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35 args += self._conf['options']
339
0ab712eab0fb cleancss: Fix stupid bug.
Ludovic Chabant <ludovic@chabant.com>
parents: 315
diff changeset
36 args.append(path)
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 logger.debug("Cleaning CSS file: %s" % args)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 try:
1112
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
40 retcode = subprocess.call(args)
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 except FileNotFoundError as ex:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 logger.error("Tried running CleanCSS processor with command: %s" %
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 args)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 raise Exception("Error running CleanCSS processor. "
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 "Did you install it?") from ex
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 if retcode != 0:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 raise Exception("Error occured in CleanCSS. Please check "
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 "log messages above for more information.")
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 return True
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 def _ensureInitialized(self):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 if self._conf is not None:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 return
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54
1112
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
55 bin_name = 'cleancss'
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
56 if platform.system() == 'Windows':
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
57 bin_name += '.cmd'
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
58
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 self._conf = self.app.config.get('cleancss') or {}
1112
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
60 self._conf.setdefault('bin', bin_name)
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61 self._conf.setdefault('options', ['--skip-rebase'])
315
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
62 self._conf.setdefault('out_ext', '.css')
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
63 if len(self._conf['out_ext']) > 0 and self._conf['out_ext'][0] != '.':
d1490028e211 cleancss: Add option to specify an output extension, like `.min.css`.
Ludovic Chabant <ludovic@chabant.com>
parents: 206
diff changeset
64 self._conf['out_ext'] = '.' + self._conf['out_ext']
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 if not isinstance(self._conf['options'], list):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66 raise Exception("The `cleancss/options` configuration setting "
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67 "must be an array of arguments.")
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70 class UglifyJSProcessor(SimpleFileProcessor):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 PROCESSOR_NAME = 'uglifyjs'
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 def __init__(self):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 super(UglifyJSProcessor, self).__init__({'js': 'js'})
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 self._conf = None
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 339
diff changeset
77 def matches(self, path):
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 339
diff changeset
78 return path.endswith('.js') and not path.endswith('.min.js')
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 339
diff changeset
79
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 def _doProcess(self, in_path, out_path):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81 self._ensureInitialized()
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83 args = [self._conf['bin'], in_path, '-o', out_path]
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84 args += self._conf['options']
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
85 logger.debug("Uglifying JS file: %s" % args)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
86
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
87 try:
1112
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
88 retcode = subprocess.call(args)
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
89 except FileNotFoundError as ex:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
90 logger.error("Tried running UglifyJS processor with command: %s" %
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
91 args)
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
92 raise Exception("Error running UglifyJS processor. "
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
93 "Did you install it?") from ex
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
94 if retcode != 0:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
95 raise Exception("Error occured in UglifyJS. Please check "
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
96 "log messages above for more information.")
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
97 return True
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
99 def _ensureInitialized(self):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
100 if self._conf is not None:
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
101 return
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
102
1112
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
103 bin_name = 'uglifyjs'
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
104 if platform.system() == 'Windows':
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
105 bin_name += '.cmd'
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
106
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
107 self._conf = self.app.config.get('uglifyjs') or {}
1112
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
108 self._conf.setdefault('bin', bin_name)
206
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
109 self._conf.setdefault('options', ['--compress'])
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
110 if not isinstance(self._conf['options'], list):
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
111 raise Exception("The `uglify/options` configuration setting "
cba781477bd0 processing: Add `concat`, `uglifyjs` and `cleancss` processors.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
112 "must be an array of arguments.")