Mercurial > piecrust2

annotate piecrust/processing/less.py @ 1188:a7c43131d871

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 15 Jun 2021 22:36:23 -0700
parents fac4483867a5
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
3 import sys
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import json
1155
fac4483867a5 less: Fix issues moving the map file on Windows. Again.
Ludovic Chabant <ludovic@chabant.com>
parents: 1112
diff changeset
5 import shutil
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 import hashlib
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7 import logging
17
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
8 import platform
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 import subprocess
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
10 from piecrust.processing.base import (
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 498
diff changeset
11 SimpleFileProcessor, ExternalProcessException, FORCE_BUILD)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 class LessProcessor(SimpleFileProcessor):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 PROCESSOR_NAME = 'less'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 super(LessProcessor, self).__init__({'less': 'css'})
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22 self._conf = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 self._map_dir = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 498
diff changeset
25 def onPipelineStart(self, ctx):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 498
diff changeset
26 self._map_dir = os.path.join(ctx.tmp_dir, 'less')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 498
diff changeset
27 if (ctx.is_main_process and
414
c4b3a7fd2f87 bake: Make pipeline processing multi-process.
Ludovic Chabant <ludovic@chabant.com>
parents: 240
diff changeset
28 not os.path.isdir(self._map_dir)):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29 os.makedirs(self._map_dir)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 def getDependencies(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 map_path = self._getMapPath(path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 try:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 with open(map_path, 'r') as f:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35 dep_map = json.load(f)
498
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
36 except OSError:
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 # Map file not found... rebuild.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 logger.debug("No map file found for LESS file '%s' at '%s'. "
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 "Rebuilding" % (path, map_path))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 return FORCE_BUILD
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41
498
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
42 # Check the version, since the `sources` list has changed
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
43 # meanings over time.
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
44 if dep_map.get('version') != 3:
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
45 logger.warning("Unknown LESS map version. Force rebuilding.")
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
46 return FORCE_BUILD
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
47
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
48 # Get the sources, but make all paths absolute.
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
49 sources = dep_map.get('sources')
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
50 path_dir = os.path.dirname(path)
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
51
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
52 def _makeAbs(p):
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
53 return os.path.join(path_dir, p)
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
54 deps = list(map(_makeAbs, sources))
1048
63be34ce6e65 bake: Don't add outputs as dependencies, duh.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
55 return deps
498
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
56
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 def _doProcess(self, in_path, out_path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 self._ensureInitialized()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60 map_path = self._getMapPath(in_path)
498
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
61 map_url = '/' + os.path.relpath(
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 498
diff changeset
62 map_path, self.app.root_dir).replace('\\', '/')
498
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
63
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
64 # On Windows, it looks like LESSC is confused with paths when the
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
65 # map file is not to be created in the same directory as the input
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
66 # file (it ends up writing invalid dependencies in the map file, with
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
67 # a mix of relative and absolute paths stuck together).
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
68 # So create it there and move it afterwards... :(
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
69 temp_map_path = os.path.join(
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 498
diff changeset
70 os.path.dirname(in_path),
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 498
diff changeset
71 os.path.basename(map_path))
498
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
72
240
c1d4e86a3918 less: Generate a proper, available URL for the LESS CSS map file.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
73 args = [self._conf['bin'],
498
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
74 '--source-map=%s' % temp_map_path,
240
c1d4e86a3918 less: Generate a proper, available URL for the LESS CSS map file.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
75 '--source-map-url=%s' % map_url]
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 args += self._conf['options']
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 args.append(in_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 args.append(out_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 logger.debug("Processing LESS file: %s" % args)
17
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
80
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
81 try:
1112
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 1048
diff changeset
82 proc = subprocess.Popen(args, stderr=subprocess.PIPE)
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
83 stdout_data, stderr_data = proc.communicate()
17
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
84 except FileNotFoundError as ex:
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
85 logger.error("Tried running LESS processor with command: %s" %
195
b4724e577a8c cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 119
diff changeset
86 args)
17
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
87 raise Exception("Error running LESS processor. "
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
88 "Did you install it?") from ex
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
89 if proc.returncode != 0:
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
90 raise ExternalProcessException(
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 498
diff changeset
91 stderr_data.decode(sys.stderr.encoding))
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
92
498
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
93 logger.debug("Moving map file: %s -> %s" % (temp_map_path, map_path))
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
94 if os.path.exists(map_path):
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
95 os.remove(map_path)
1155
fac4483867a5 less: Fix issues moving the map file on Windows. Again.
Ludovic Chabant <ludovic@chabant.com>
parents: 1112
diff changeset
96 shutil.move(temp_map_path, map_path)
498
35d4c172e5a6 less: Fix issues with the map file on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 414
diff changeset
97
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98 return True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
99
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
100 def _ensureInitialized(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
101 if self._conf is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
102 return
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
103
1112
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 1048
diff changeset
104 bin_name = 'lessc'
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 1048
diff changeset
105 if platform.system() == 'Windows':
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 1048
diff changeset
106 bin_name += '.cmd'
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 1048
diff changeset
107
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
108 self._conf = self.app.config.get('less') or {}
1112
a3dec0fbd9ce bake: Fix bug on Windows where shim scripts of NodeJS tools couldn't run.
Ludovic Chabant <ludovic@chabant.com>
parents: 1048
diff changeset
109 self._conf.setdefault('bin', bin_name)
195
b4724e577a8c cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 119
diff changeset
110 self._conf.setdefault('options', ['--compress'])
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
111 if not isinstance(self._conf['options'], list):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
112 raise Exception("The `less/options` configuration setting "
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
113 "must be an array of arguments.")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
114
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
115 def _getMapPath(self, path):
119
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
116 map_name = "%s_%s.map" % (
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 498
diff changeset
117 os.path.basename(path),
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 498
diff changeset
118 hashlib.md5(path.encode('utf8')).hexdigest())
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
119 map_path = os.path.join(self._map_dir, map_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
120 return map_path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
121