Mercurial > piecrust2

annotate piecrust/data/paginationdata.py @ 1188:a7c43131d871

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 15 Jun 2021 22:36:23 -0700
parents 7e4742a60d14
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
912
cd236a6af9f6 data: Add access to route metadata in the templating data.
Ludovic Chabant <ludovic@chabant.com>
parents: 830
diff changeset
1 import copy
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import time
718
d4408fbbbc7d internal: Prevent crash because of missing logger.
Ludovic Chabant <ludovic@chabant.com>
parents: 711
diff changeset
3 import logging
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 from piecrust.data.pagedata import LazyPageConfigData
877
d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps.
Ludovic Chabant <ludovic@chabant.com>
parents: 874
diff changeset
5 from piecrust.sources.base import AbortedSourceUseError
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
718
d4408fbbbc7d internal: Prevent crash because of missing logger.
Ludovic Chabant <ludovic@chabant.com>
parents: 711
diff changeset
8 logger = logging.getLogger(__name__)
d4408fbbbc7d internal: Prevent crash because of missing logger.
Ludovic Chabant <ludovic@chabant.com>
parents: 711
diff changeset
9
d4408fbbbc7d internal: Prevent crash because of missing logger.
Ludovic Chabant <ludovic@chabant.com>
parents: 711
diff changeset
10
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11 class PaginationData(LazyPageConfigData):
1055
7e4742a60d14 data: Add new properties and folder support to the `family` data endpoint.
Ludovic Chabant <ludovic@chabant.com>
parents: 979
diff changeset
12 def __init__(self, page, extra_data=None):
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
13 super().__init__(page)
1055
7e4742a60d14 data: Add new properties and folder support to the `family` data endpoint.
Ludovic Chabant <ludovic@chabant.com>
parents: 979
diff changeset
14 if extra_data:
7e4742a60d14 data: Add new properties and folder support to the `family` data endpoint.
Ludovic Chabant <ludovic@chabant.com>
parents: 979
diff changeset
15 self._values.update(extra_data)
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 def _load(self):
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 830
diff changeset
18 from piecrust.uriutil import split_uri
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 830
diff changeset
19
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 page = self._page
874
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
21 set_val = self._setValue
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
22
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
23 page_url = page.getUri()
912
cd236a6af9f6 data: Add access to route metadata in the templating data.
Ludovic Chabant <ludovic@chabant.com>
parents: 830
diff changeset
24 _, rel_url = split_uri(page.app, page_url)
874
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
25 set_val('url', page_url)
922
b447c24bc8d4 Merge changes from PieCrust2 branch.
Ludovic Chabant <ludovic@chabant.com>
parents: 877 912
diff changeset
26 set_val('rel_url', rel_url)
b447c24bc8d4 Merge changes from PieCrust2 branch.
Ludovic Chabant <ludovic@chabant.com>
parents: 877 912
diff changeset
27 set_val('slug', rel_url) # For backwards compatibility
b447c24bc8d4 Merge changes from PieCrust2 branch.
Ludovic Chabant <ludovic@chabant.com>
parents: 877 912
diff changeset
28 set_val('route', copy.deepcopy(page.source_metadata['route_params']))
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29
874
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
30 self._mapLoader('date', _load_date)
966
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
31 self._mapLoader('datetime', _load_datetime)
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
32 self._mapLoader('timestamp', _load_timestamp)
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
33 self._mapLoader('mtime', _load_content_mtime)
874
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
34 self._mapLoader('assets', _load_assets)
1055
7e4742a60d14 data: Add new properties and folder support to the `family` data endpoint.
Ludovic Chabant <ludovic@chabant.com>
parents: 979
diff changeset
35 self._mapLoader('family', _load_family)
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 segment_names = page.config.get('segments')
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 for name in segment_names:
966
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
39 self._mapLoader('raw_' + name, _load_raw_segment)
874
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
40 self._mapLoader(name, _load_rendered_segment)
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
41
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
42
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
43 def _load_assets(data, name):
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
44 from piecrust.data.assetor import Assetor
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
45 return Assetor(data._page)
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
46
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47
1055
7e4742a60d14 data: Add new properties and folder support to the `family` data endpoint.
Ludovic Chabant <ludovic@chabant.com>
parents: 979
diff changeset
48 def _load_family(data, name):
7e4742a60d14 data: Add new properties and folder support to the `family` data endpoint.
Ludovic Chabant <ludovic@chabant.com>
parents: 979
diff changeset
49 from piecrust.data.linker import Linker
7e4742a60d14 data: Add new properties and folder support to the `family` data endpoint.
Ludovic Chabant <ludovic@chabant.com>
parents: 979
diff changeset
50 return Linker(data._page.source, data._page.content_item)
7e4742a60d14 data: Add new properties and folder support to the `family` data endpoint.
Ludovic Chabant <ludovic@chabant.com>
parents: 979
diff changeset
51
7e4742a60d14 data: Add new properties and folder support to the `family` data endpoint.
Ludovic Chabant <ludovic@chabant.com>
parents: 979
diff changeset
52
874
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
53 def _load_date(data, name):
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
54 page = data._page
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
55 date_format = page.app.config.get('site/date_format')
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
56 if date_format:
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
57 return page.datetime.strftime(date_format)
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
58 return None
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
59
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60
966
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
61 def _load_datetime(data, name):
979
45ad976712ec tests: Big push to get the tests to pass again.
Ludovic Chabant <ludovic@chabant.com>
parents: 966
diff changeset
62 dt = data._page.datetime
966
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
63 return {
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
64 'year': dt.year, 'month': dt.month, 'day': dt.day,
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
65 'hour': dt.hour, 'minute': dt.minute, 'second': dt.second}
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
66
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
67
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
68 def _load_timestamp(data, name):
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
69 page = data._page
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
70 return time.mktime(page.datetime.timetuple())
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
71
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
72
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
73 def _load_content_mtime(data, name):
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
74 return data._page.content_mtime
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
75
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
76
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
77 def _load_raw_segment(data, name):
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
78 page = data._page
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
79 return page.getSegment(name[4:])
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
80
cc55740860de data: Delay loading page configuration and datetimes.
Ludovic Chabant <ludovic@chabant.com>
parents: 922
diff changeset
81
874
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
82 def _load_rendered_segment(data, name):
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
83 page = data._page
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
84
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
85 do_render = True
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
86 stack = page.app.env.render_ctx_stack
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
87 if stack.hasPage(page):
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
88 # This is the pagination data for the page that is currently
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
89 # being rendered! Inception! But this is possible... so just
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
90 # prevent infinite recursion.
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
91 do_render = False
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
92
874
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
93 if do_render:
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
94 uri = page.getUri()
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
95 try:
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
96 from piecrust.rendering import (
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
97 RenderingContext, render_page_segments)
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
98 ctx = RenderingContext(page)
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
99 render_result = render_page_segments(ctx)
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
100 segs = render_result.segments
877
d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps.
Ludovic Chabant <ludovic@chabant.com>
parents: 874
diff changeset
101 except AbortedSourceUseError:
d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps.
Ludovic Chabant <ludovic@chabant.com>
parents: 874
diff changeset
102 raise
874
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
103 except Exception as ex:
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
104 logger.exception(ex)
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
105 raise Exception(
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
106 "Error rendering segments for '%s'" % uri) from ex
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
107 else:
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
108 segs = {}
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
109 for name in page.config.get('segments'):
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
110 segs[name] = "<unavailable: current page>"
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
111
874
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
112 unmap_loader = data._unmapLoader
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
113 set_val = data._setValue
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
114
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
115 for k, v in segs.items():
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
116 unmap_loader(k)
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
117 set_val(k, v)
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
118
874
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
119 if 'content.abstract' in segs:
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
120 set_val('content', segs['content.abstract'])
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
121 set_val('has_more', True)
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
122 if name == 'content':
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
123 return segs['content.abstract']
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
124
874
f4608e2e80ce data: Optimize page data creation.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
125 return segs[name]
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
126