Mercurial > piecrust2

annotate piecrust/data/assetor.py @ 411:e7b865f8f335

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.
author Ludovic Chabant <ludovic@chabant.com>
date Fri, 12 Jun 2015 17:09:19 -0700
parents 422052d2e978
children 0e9a94b7fdfa
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
1 import os
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import logging
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 from piecrust.uriutil import multi_replace
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
10 class UnsupportedAssetsError(Exception):
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
11 pass
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
12
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
13
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 25
diff changeset
14 def build_base_url(app, uri, rel_assets_path):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 base_url_format = app.env.base_asset_url_format
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 25
diff changeset
16 rel_assets_path = rel_assets_path.replace('\\', '/')
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 25
diff changeset
17
33
62c7a97c8340 Get the un-paginated URL of a page early and pass that around.
Ludovic Chabant <ludovic@chabant.com>
parents: 32
diff changeset
18 # Remove any extension since we'll be copying assets into the 1st
62c7a97c8340 Get the un-paginated URL of a page early and pass that around.
Ludovic Chabant <ludovic@chabant.com>
parents: 32
diff changeset
19 # sub-page's folder.
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 pretty = app.config.get('site/pretty_urls')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 if not pretty:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22 uri, _ = os.path.splitext(uri)
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 25
diff changeset
23
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 base_url = multi_replace(
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 base_url_format,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 {
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 25
diff changeset
27 '%path%': rel_assets_path,
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 '%uri%': uri})
329
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 290
diff changeset
29
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 290
diff changeset
30 return base_url.rstrip('/') + '/'
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 class Assetor(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 ASSET_DIR_SUFFIX = '-assets'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 debug_render_doc = """Helps render URLs to files in the current page's
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 asset folder."""
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 debug_render = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 debug_render_dynamic = ['_debugRenderAssetNames']
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 def __init__(self, page, uri):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 self._page = page
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 self._uri = uri
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 self._cache = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 def __getattr__(self, name):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 try:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 self._cacheAssets()
25
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
49 return self._cache[name][0]
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 except KeyError:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 raise AttributeError()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 def __getitem__(self, key):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 self._cacheAssets()
25
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
55 return self._cache[key][0]
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 def __iter__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 self._cacheAssets()
25
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
59 return map(lambda i: i[0], self._cache.values())
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
60
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61 def _debugRenderAssetNames(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62 self._cacheAssets()
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
63 return list(self._cache.keys())
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 def _cacheAssets(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66 if self._cache is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67 return
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 self._cache = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70 name, ext = os.path.splitext(self._page.path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 assets_dir = name + Assetor.ASSET_DIR_SUFFIX
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 if not os.path.isdir(assets_dir):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 return
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 25
diff changeset
75 rel_assets_dir = os.path.relpath(assets_dir, self._page.app.root_dir)
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 25
diff changeset
76 base_url = build_base_url(self._page.app, self._uri, rel_assets_dir)
25
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
77 for fn in os.listdir(assets_dir):
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
78 full_fn = os.path.join(assets_dir, fn)
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
79 if not os.path.isfile(full_fn):
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
80 raise Exception("Skipping: %s" % full_fn)
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
81 continue
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82
25
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
83 name, ext = os.path.splitext(fn)
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
84 if name in self._cache:
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
85 raise UnsupportedAssetsError(
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
86 "Multiple asset files are named '%s'." % name)
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
87 self._cache[name] = (base_url + fn, full_fn)
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
88
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
89 cpi = self._page.app.env.exec_info_stack.current_page_info
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
90 if cpi is not None:
96
0445a2232de7 Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 33
diff changeset
91 used_assets = list(map(lambda i: i[1], self._cache.values()))
0445a2232de7 Improvements and fixes to incremental baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 33
diff changeset
92 cpi.render_ctx.used_assets = used_assets
25
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
93