Mercurial > piecrust2

annotate piecrust/processing/less.py @ 411:e7b865f8f335

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.
author Ludovic Chabant <ludovic@chabant.com>
date Fri, 12 Jun 2015 17:09:19 -0700
parents c1d4e86a3918
children c4b3a7fd2f87
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
3 import sys
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import json
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import hashlib
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 import logging
17
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
7 import platform
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 import subprocess
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
9 from piecrust.processing.base import (
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
10 SimpleFileProcessor, ExternalProcessException)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11 from piecrust.processing.tree import FORCE_BUILD
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 class LessProcessor(SimpleFileProcessor):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 PROCESSOR_NAME = 'less'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 super(LessProcessor, self).__init__({'less': 'css'})
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22 self._conf = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 self._map_dir = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 def onPipelineStart(self, pipeline):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 self._map_dir = os.path.join(pipeline.tmp_dir, 'less')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 if not os.path.isdir(self._map_dir):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 os.makedirs(self._map_dir)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 def getDependencies(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 map_path = self._getMapPath(path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 try:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 with open(map_path, 'r') as f:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 dep_map = json.load(f)
119
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
35
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
36 # Check the version, since the `sources` list has changed
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
37 # meanings over time.
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
38 if dep_map.get('version') != 3:
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
39 logger.warning("Unknown LESS map version. Force rebuilding.")
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
40 return FORCE_BUILD
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
41
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
42 # Get the sources, but make all paths absolute.
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
43 sources = dep_map.get('sources')
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 path_dir = os.path.dirname(path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 def _makeAbs(p):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 return os.path.join(path_dir, p)
119
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
47 deps = list(map(_makeAbs, sources))
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
48 return [map_path] + deps
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 except IOError:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 # Map file not found... rebuild.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 logger.debug("No map file found for LESS file '%s' at '%s'. "
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 "Rebuilding" % (path, map_path))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 return FORCE_BUILD
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 def _doProcess(self, in_path, out_path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56 self._ensureInitialized()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 map_path = self._getMapPath(in_path)
240
c1d4e86a3918 less: Generate a proper, available URL for the LESS CSS map file.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
59 map_url = '/' + os.path.relpath(map_path, self.app.root_dir)
c1d4e86a3918 less: Generate a proper, available URL for the LESS CSS map file.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
60 args = [self._conf['bin'],
c1d4e86a3918 less: Generate a proper, available URL for the LESS CSS map file.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
61 '--source-map=%s' % map_path,
c1d4e86a3918 less: Generate a proper, available URL for the LESS CSS map file.
Ludovic Chabant <ludovic@chabant.com>
parents: 221
diff changeset
62 '--source-map-url=%s' % map_url]
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63 args += self._conf['options']
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 args.append(in_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 args.append(out_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66 logger.debug("Processing LESS file: %s" % args)
17
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
67
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
68 # On Windows, we need to run the process in a shell environment
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
69 # otherwise it looks like `PATH` isn't taken into account.
116
1c13f3389fcb Cosmetic fix.
Ludovic Chabant <ludovic@chabant.com>
parents: 17
diff changeset
70 shell = (platform.system() == 'Windows')
17
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
71 try:
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
72 proc = subprocess.Popen(
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
73 args, shell=shell,
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
74 stderr=subprocess.PIPE)
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
75 stdout_data, stderr_data = proc.communicate()
17
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
76 except FileNotFoundError as ex:
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
77 logger.error("Tried running LESS processor with command: %s" %
195
b4724e577a8c cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 119
diff changeset
78 args)
17
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
79 raise Exception("Error running LESS processor. "
de6a296744f7 The LESS compiler must be launched in a shell on Windows.
Ludovic Chabant <ludovic@chabant.com>
parents: 7
diff changeset
80 "Did you install it?") from ex
221
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
81 if proc.returncode != 0:
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
82 raise ExternalProcessException(
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
83 stderr_data.decode(sys.stderr.encoding))
f82262f59600 bake: Fix processing record bugs and error logging for external processes.
Ludovic Chabant <ludovic@chabant.com>
parents: 195
diff changeset
84
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
85 return True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
86
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
87 def _ensureInitialized(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
88 if self._conf is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
89 return
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
90
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
91 self._conf = self.app.config.get('less') or {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
92 self._conf.setdefault('bin', 'lessc')
195
b4724e577a8c cosmetic: Fix some PEP8 issues.
Ludovic Chabant <ludovic@chabant.com>
parents: 119
diff changeset
93 self._conf.setdefault('options', ['--compress'])
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
94 if not isinstance(self._conf['options'], list):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
95 raise Exception("The `less/options` configuration setting "
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
96 "must be an array of arguments.")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
97
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98 def _getMapPath(self, path):
119
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
99 map_name = "%s_%s.map" % (
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
100 os.path.basename(path),
0811f92cbdc7 Slightly more robust dependency handling for the LESS processor.
Ludovic Chabant <ludovic@chabant.com>
parents: 116
diff changeset
101 hashlib.md5(path.encode('utf8')).hexdigest())
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
102 map_path = os.path.join(self._map_dir, map_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
103 return map_path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
104