annotate piecrust/data/assetor.py @ 1188:a7c43131d871

bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 15 Jun 2021 22:36:23 -0700
parents 0aeb6f18d6a0
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
1 import os
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
1108
b2a34a6ec5e5 data: Let the `asset` endpoint load JSON data into the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
3 import json
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import logging
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 837
diff changeset
5 from piecrust.sources.base import REL_ASSETS
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 from piecrust.uriutil import multi_replace
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
12 class UnsupportedAssetsError(Exception):
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
13 pass
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
14
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
15
862
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
16 class _AssetInfo:
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
17 def __init__(self, content_item, uri):
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
18 self.content_item = content_item
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
19 self.uri = uri
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20
1108
b2a34a6ec5e5 data: Let the `asset` endpoint load JSON data into the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
21 def __str__(self):
b2a34a6ec5e5 data: Let the `asset` endpoint load JSON data into the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
22 return self.uri
b2a34a6ec5e5 data: Let the `asset` endpoint load JSON data into the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
23
1113
29c51b981c17 data: Rename JSON-ifier function.
Ludovic Chabant <ludovic@chabant.com>
parents: 1108
diff changeset
24 def as_json(self):
1108
b2a34a6ec5e5 data: Let the `asset` endpoint load JSON data into the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
25 with open(self.content_item.spec, 'r', encoding='utf8') as fp:
b2a34a6ec5e5 data: Let the `asset` endpoint load JSON data into the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
26 return json.load(fp)
b2a34a6ec5e5 data: Let the `asset` endpoint load JSON data into the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
27
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28
915
9d804fa186a0 data: Make the `Assetor` more into a `dict` than a `list`.
Ludovic Chabant <ludovic@chabant.com>
parents: 886
diff changeset
29 class Assetor:
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 837
diff changeset
30 debug_render_doc = """Helps render URLs to files in the current page's
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 837
diff changeset
31 asset folder."""
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 837
diff changeset
32 debug_render = []
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 837
diff changeset
33 debug_render_dynamic = ['_debugRenderAssetNames']
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 837
diff changeset
34
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
35 def __init__(self, page):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 self._page = page
867
757fba54bfd3 refactor: Improve pagination and iterators to work with other sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 862
diff changeset
37 self._cache_map = None
757fba54bfd3 refactor: Improve pagination and iterators to work with other sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 862
diff changeset
38 self._cache_list = None
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 def __getattr__(self, name):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 try:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 self._cacheAssets()
1108
b2a34a6ec5e5 data: Let the `asset` endpoint load JSON data into the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
43 return self._cache_map[name]
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 except KeyError:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 raise AttributeError()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46
915
9d804fa186a0 data: Make the `Assetor` more into a `dict` than a `list`.
Ludovic Chabant <ludovic@chabant.com>
parents: 886
diff changeset
47 def __getitem__(self, name):
9d804fa186a0 data: Make the `Assetor` more into a `dict` than a `list`.
Ludovic Chabant <ludovic@chabant.com>
parents: 886
diff changeset
48 self._cacheAssets()
1108
b2a34a6ec5e5 data: Let the `asset` endpoint load JSON data into the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
49 return self._cache_map[name]
915
9d804fa186a0 data: Make the `Assetor` more into a `dict` than a `list`.
Ludovic Chabant <ludovic@chabant.com>
parents: 886
diff changeset
50
9d804fa186a0 data: Make the `Assetor` more into a `dict` than a `list`.
Ludovic Chabant <ludovic@chabant.com>
parents: 886
diff changeset
51 def __contains__(self, name):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 self._cacheAssets()
915
9d804fa186a0 data: Make the `Assetor` more into a `dict` than a `list`.
Ludovic Chabant <ludovic@chabant.com>
parents: 886
diff changeset
53 return name in self._cache_map
9d804fa186a0 data: Make the `Assetor` more into a `dict` than a `list`.
Ludovic Chabant <ludovic@chabant.com>
parents: 886
diff changeset
54
9d804fa186a0 data: Make the `Assetor` more into a `dict` than a `list`.
Ludovic Chabant <ludovic@chabant.com>
parents: 886
diff changeset
55 def __iter__(self):
9d804fa186a0 data: Make the `Assetor` more into a `dict` than a `list`.
Ludovic Chabant <ludovic@chabant.com>
parents: 886
diff changeset
56 self._cacheAssets()
9d804fa186a0 data: Make the `Assetor` more into a `dict` than a `list`.
Ludovic Chabant <ludovic@chabant.com>
parents: 886
diff changeset
57 return iter(self._cache_list)
25
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 13
diff changeset
58
862
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
59 def __len__(self):
809
22c6f6a3d0a0 admin: Add ability to upload page assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 472
diff changeset
60 self._cacheAssets()
915
9d804fa186a0 data: Make the `Assetor` more into a `dict` than a `list`.
Ludovic Chabant <ludovic@chabant.com>
parents: 886
diff changeset
61 return len(self._cache_map)
415
0e9a94b7fdfa bake: Improve bake record information.
Ludovic Chabant <ludovic@chabant.com>
parents: 329
diff changeset
62
886
dcdec4b951a1 admin: Get the admin panel working again.
Ludovic Chabant <ludovic@chabant.com>
parents: 871
diff changeset
63 def _getAssetNames(self):
dcdec4b951a1 admin: Get the admin panel working again.
Ludovic Chabant <ludovic@chabant.com>
parents: 871
diff changeset
64 self._cacheAssets()
dcdec4b951a1 admin: Get the admin panel working again.
Ludovic Chabant <ludovic@chabant.com>
parents: 871
diff changeset
65 return self._cache_map.keys()
dcdec4b951a1 admin: Get the admin panel working again.
Ludovic Chabant <ludovic@chabant.com>
parents: 871
diff changeset
66
871
504ddb370df8 refactor: Fixing some issues with baking assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 870
diff changeset
67 def _getAssetItems(self):
504ddb370df8 refactor: Fixing some issues with baking assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 870
diff changeset
68 self._cacheAssets()
504ddb370df8 refactor: Fixing some issues with baking assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 870
diff changeset
69 return map(lambda i: i.content_item, self._cache_map.values())
504ddb370df8 refactor: Fixing some issues with baking assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 870
diff changeset
70
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 def _debugRenderAssetNames(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 self._cacheAssets()
870
48d25fd68b8d assets: Fix bug in assetor.
Ludovic Chabant <ludovic@chabant.com>
parents: 867
diff changeset
73 return list(self._cache_map.keys())
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 def _cacheAssets(self):
867
757fba54bfd3 refactor: Improve pagination and iterators to work with other sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 862
diff changeset
76 if self._cache_map is not None:
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 return
837
ad8f48a31c62 assets: Fix crash when a page doesn't have assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 832
diff changeset
78
862
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
79 source = self._page.source
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 837
diff changeset
80 content_item = self._page.content_item
1148
0aeb6f18d6a0 bake: Don't crash if a source doesn't implement related items.
Ludovic Chabant <ludovic@chabant.com>
parents: 1113
diff changeset
81 try:
0aeb6f18d6a0 bake: Don't crash if a source doesn't implement related items.
Ludovic Chabant <ludovic@chabant.com>
parents: 1113
diff changeset
82 assets = source.getRelatedContents(content_item, REL_ASSETS)
0aeb6f18d6a0 bake: Don't crash if a source doesn't implement related items.
Ludovic Chabant <ludovic@chabant.com>
parents: 1113
diff changeset
83 except NotImplementedError:
0aeb6f18d6a0 bake: Don't crash if a source doesn't implement related items.
Ludovic Chabant <ludovic@chabant.com>
parents: 1113
diff changeset
84 assets = None
862
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
85
867
757fba54bfd3 refactor: Improve pagination and iterators to work with other sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 862
diff changeset
86 self._cache_map = {}
757fba54bfd3 refactor: Improve pagination and iterators to work with other sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 862
diff changeset
87 self._cache_list = []
757fba54bfd3 refactor: Improve pagination and iterators to work with other sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 862
diff changeset
88
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 837
diff changeset
89 if assets is None:
862
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
90 return
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
91
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 837
diff changeset
92 app = source.app
862
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
93 root_dir = app.root_dir
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
94 asset_url_format = app.config.get('site/asset_url_format')
975
a0a62d0da723 internal: Check that the `Assetor` has an asset URL format to work with.
Ludovic Chabant <ludovic@chabant.com>
parents: 915
diff changeset
95 if not asset_url_format:
a0a62d0da723 internal: Check that the `Assetor` has an asset URL format to work with.
Ludovic Chabant <ludovic@chabant.com>
parents: 915
diff changeset
96 raise Exception("No asset URL format was specified.")
862
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
97
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
98 page_uri = self._page.getUri()
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
99 pretty_urls = app.config.get('site/pretty_urls')
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
100 if not pretty_urls:
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
101 page_uri, _ = os.path.splitext(page_uri)
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
102
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
103 uri_build_tokens = {
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
104 '%path%': None,
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
105 '%filename%': None,
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
106 '%page_uri%': page_uri
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
107 }
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
108
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
109 for a in assets:
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
110 name = a.metadata['name']
867
757fba54bfd3 refactor: Improve pagination and iterators to work with other sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 862
diff changeset
111 if name in self._cache_map:
862
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
112 raise UnsupportedAssetsError(
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
113 "An asset with name '%s' already exists for item '%s'. "
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
114 "Do you have multiple assets with colliding names?" %
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
115 (name, content_item.spec))
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
116
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
117 # TODO: this assumes a file-system source!
871
504ddb370df8 refactor: Fixing some issues with baking assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 870
diff changeset
118 uri_build_tokens['%path%'] = \
504ddb370df8 refactor: Fixing some issues with baking assets.
Ludovic Chabant <ludovic@chabant.com>
parents: 870
diff changeset
119 os.path.relpath(a.spec, root_dir).replace('\\', '/')
1108
b2a34a6ec5e5 data: Let the `asset` endpoint load JSON data into the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
120 uri_build_tokens['%filename%'] = a.metadata.get('filename')
862
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
121 uri = multi_replace(asset_url_format, uri_build_tokens)
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
122
867
757fba54bfd3 refactor: Improve pagination and iterators to work with other sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 862
diff changeset
123 self._cache_map[name] = _AssetInfo(a, uri)
757fba54bfd3 refactor: Improve pagination and iterators to work with other sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 862
diff changeset
124 self._cache_list.append(uri)
862
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
125
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 837
diff changeset
126 stack = app.env.render_ctx_stack
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 837
diff changeset
127 cur_ctx = stack.current_ctx
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 837
diff changeset
128 if cur_ctx is not None:
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 975
diff changeset
129 cur_ctx.render_info['used_assets'] = True
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
130