Mercurial > piecrust2

annotate piecrust/data/builder.py @ 1188:a7c43131d871

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 15 Jun 2021 22:36:23 -0700
parents 501bd4ab7e06
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import logging
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
2 from piecrust.data.assetor import Assetor
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 420
diff changeset
3 from piecrust.data.base import MergedMapping
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
4 from piecrust.data.linker import Linker
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 420
diff changeset
5 from piecrust.data.pagedata import PageData
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 from piecrust.data.paginator import Paginator
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 420
diff changeset
7 from piecrust.data.piecrustdata import PieCrustData
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 420
diff changeset
8 from piecrust.data.providersdata import DataProvidersData
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
9 from piecrust.routing import RouteFunction
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
15 class DataBuildingContext:
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
16 def __init__(self, page, sub_num):
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
17 self.page = page
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
18 self.sub_num = sub_num
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19 self.pagination_source = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 self.pagination_filter = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 def build_page_data(ctx):
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
24 page = ctx.page
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
25 sub_num = ctx.sub_num
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
26 app = page.app
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
27
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 420
diff changeset
28 pgn_source = ctx.pagination_source or get_default_pagination_source(page)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29
12
30a42341cfa8 Define page slugs properly, avoid recursions with debug data.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
30 pc_data = PieCrustData()
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 420
diff changeset
31 config_data = PageData(page, ctx)
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
32 paginator = Paginator(pgn_source, page, sub_num,
369
4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 352
diff changeset
33 pgn_filter=ctx.pagination_filter)
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
34 assetor = Assetor(page)
866
d9059257743c refactor: Make the linker work again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
35 linker = Linker(page.source, page.content_item)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 data = {
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
37 'piecrust': pc_data,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
38 'page': config_data,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
39 'assets': assetor,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
40 'pagination': paginator,
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
41 'family': linker
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
42 }
420
f1b759c188b0 internal: Optimize page data building.
Ludovic Chabant <ludovic@chabant.com>
parents: 406
diff changeset
43
802
0da1207472d3 templating: Put the routing functions in the data, not the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 723
diff changeset
44 for route in app.routes:
0da1207472d3 templating: Put the routing functions in the data, not the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 723
diff changeset
45 name = route.func_name
847
71c4f43d8fc1 data: Don't add route functions or data providers that happen to be null.
Ludovic Chabant <ludovic@chabant.com>
parents: 830
diff changeset
46 if not name:
71c4f43d8fc1 data: Don't add route functions or data providers that happen to be null.
Ludovic Chabant <ludovic@chabant.com>
parents: 830
diff changeset
47 continue
71c4f43d8fc1 data: Don't add route functions or data providers that happen to be null.
Ludovic Chabant <ludovic@chabant.com>
parents: 830
diff changeset
48
802
0da1207472d3 templating: Put the routing functions in the data, not the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 723
diff changeset
49 func = data.get(name)
0da1207472d3 templating: Put the routing functions in the data, not the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 723
diff changeset
50 if func is None:
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
51 data[name] = RouteFunction(route)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
52 elif isinstance(func, RouteFunction):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
53 if not func._isCompatibleRoute(route):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
54 raise Exception(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
55 "Route function '%s' can't target both route '%s' and "
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
56 "route '%s' as the 2 patterns are incompatible." %
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
57 (name, func._route.uri_pattern, route.uri_pattern))
802
0da1207472d3 templating: Put the routing functions in the data, not the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 723
diff changeset
58 else:
0da1207472d3 templating: Put the routing functions in the data, not the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 723
diff changeset
59 raise Exception("Route function '%s' collides with an "
0da1207472d3 templating: Put the routing functions in the data, not the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 723
diff changeset
60 "existing function or template data." %
0da1207472d3 templating: Put the routing functions in the data, not the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 723
diff changeset
61 name)
0da1207472d3 templating: Put the routing functions in the data, not the template engine.
Ludovic Chabant <ludovic@chabant.com>
parents: 723
diff changeset
62
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
63 # TODO: handle slugified taxonomy terms.
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 420
diff changeset
65 site_data = app.config.getAll()
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 420
diff changeset
66 providers_data = DataProvidersData(page)
985
74f4bde40fd0 data: Optimize page data so cheapest accessors are tried first.
Ludovic Chabant <ludovic@chabant.com>
parents: 866
diff changeset
67
74f4bde40fd0 data: Optimize page data so cheapest accessors are tried first.
Ludovic Chabant <ludovic@chabant.com>
parents: 866
diff changeset
68 # Put the site data first so that `MergedMapping` doesn't load stuff
74f4bde40fd0 data: Optimize page data so cheapest accessors are tried first.
Ludovic Chabant <ludovic@chabant.com>
parents: 866
diff changeset
69 # for nothing just to find a value that was in the YAML config all
74f4bde40fd0 data: Optimize page data so cheapest accessors are tried first.
Ludovic Chabant <ludovic@chabant.com>
parents: 866
diff changeset
70 # along.
1010
501bd4ab7e06 internal: Remove unused parameter.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
71 data = MergedMapping([site_data, data, providers_data])
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 # Do this at the end because we want all the data to be ready to be
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 # displayed in the debugger window.
223
7decf00eee47 serve: Don't expose the debug info right away when running with `--debug`.
Ludovic Chabant <ludovic@chabant.com>
parents: 222
diff changeset
75 if (app.config.get('site/show_debug_info') and
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 not app.config.get('baker/is_baking')):
556
93b656f0af54 serve: Improve debug information in the preview server.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
77 pc_data.enableDebugInfo(page)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 return data
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
82 def add_layout_data(page_data, contents):
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
83 for name, txt in contents.items():
369
4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 352
diff changeset
84 if name in page_data:
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
85 logger.warning("Content segment '%s' will hide existing data." %
369
4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 352
diff changeset
86 name)
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 420
diff changeset
87 page_data._prependMapping(contents)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
88
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
89
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
90 def get_default_pagination_source(page):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
91 app = page.app
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
92 source_name = page.config.get('source') or page.config.get('blog')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
93 if source_name is None:
323
412537e91e45 pagination: Add support for `site/default_pagination_source`.
Ludovic Chabant <ludovic@chabant.com>
parents: 298
diff changeset
94 source_name = app.config.get('site/default_pagination_source')
412537e91e45 pagination: Add support for `site/default_pagination_source`.
Ludovic Chabant <ludovic@chabant.com>
parents: 298
diff changeset
95 if source_name is None:
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
96 blog_names = app.config.get('site/blogs')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
97 if blog_names is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98 source_name = blog_names[0]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
99 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
100 source_name = app.sources[0].name
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
101 source = app.getSource(source_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
102 return source
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
103