Mercurial > piecrust2

annotate piecrust/sources/default.py @ 1188:a7c43131d871

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 15 Jun 2021 22:36:23 -0700
parents 4cc020ff2537
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os.path
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import logging
792
58ebf50235a5 routing: Simplify how routes are defined.
Ludovic Chabant <ludovic@chabant.com>
parents: 577
diff changeset
3 from piecrust.routing import RouteParameter
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
4 from piecrust.sources.base import REL_ASSETS, ContentItem
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
5 from piecrust.sources.fs import FSContentSource
576
0c74a6c4533d sources: Add code to support "interactive" metadata acquisition.
Ludovic Chabant <ludovic@chabant.com>
parents: 520
diff changeset
6 from piecrust.sources.interfaces import (
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
7 IPreparingSource, IInteractiveSource, InteractiveField)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
8 from piecrust.sources.mixins import SimpleAssetsSubDirMixin
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
9 from piecrust.uriutil import uri_to_title
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12 logger = logging.getLogger(__name__)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
15 class DefaultContentSource(FSContentSource,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
16 SimpleAssetsSubDirMixin,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
17 IPreparingSource, IInteractiveSource):
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 SOURCE_NAME = 'default'
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
19 DEFAULT_PIPELINE_NAME = 'page'
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 def __init__(self, app, name, config):
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
22 super().__init__(app, name, config)
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
23
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
24 config.setdefault('data_type', 'page_iterator')
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
25
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
26 self.auto_formats = app.config.get('site/auto_formats')
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 self.default_auto_format = app.config.get('site/default_auto_format')
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
28 self.supported_extensions = list(self.auto_formats)
520
bab91fcef741 bake/serve: Improve support for unicode, add slugification options.
Ludovic Chabant <ludovic@chabant.com>
parents: 366
diff changeset
29
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
30 def _finalizeContent(self, parent_group, items, groups):
866
d9059257743c refactor: Make the linker work again.
Ludovic Chabant <ludovic@chabant.com>
parents: 862
diff changeset
31 SimpleAssetsSubDirMixin._removeAssetGroups(self, groups)
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 944
diff changeset
33 def _createItemMetadata(self, path):
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
34 slug = self._makeSlug(path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
35 metadata = {
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
36 'route_params': {
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
37 'slug': slug
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
38 }
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
39 }
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
40 _, ext = os.path.splitext(path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
41 if ext:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
42 fmt = self.auto_formats.get(ext.lstrip('.'))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
43 if fmt:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
44 metadata['config'] = {'format': fmt}
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
45 return metadata
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
47 def _makeSlug(self, path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
48 rel_path = os.path.relpath(path, self.fs_endpoint_path)
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 slug, ext = os.path.splitext(rel_path)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 slug = slug.replace('\\', '/')
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 if ext.lstrip('.') not in self.supported_extensions:
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 slug += ext
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 if slug.startswith('./'):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 slug = slug[2:]
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 if slug == '_index':
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56 slug = ''
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 return slug
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
59 def getRelatedContents(self, item, relationship):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
60 if relationship == REL_ASSETS:
862
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
61 return SimpleAssetsSubDirMixin._getRelatedAssetsContents(
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
62 self, item)
fddaf43424e2 refactor: Get the page assets to work again in the server.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
63 return FSContentSource.getRelatedContents(self, item, relationship)
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
64
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
65 def getSupportedRouteParameters(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
66 return [
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
67 RouteParameter('slug', RouteParameter.TYPE_PATH)]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
68
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 944
diff changeset
69 def findContentFromRoute(self, route_params):
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
70 uri_path = route_params.get('slug', '')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
71 if not uri_path:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
72 uri_path = '_index'
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
73 path = os.path.join(self.fs_endpoint_path, uri_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
74 _, ext = os.path.splitext(path)
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
76 if ext == '':
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
77 paths_to_check = [
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
78 '%s.%s' % (path, e)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
79 for e in self.supported_extensions]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
80 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
81 paths_to_check = [path]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
82 for path in paths_to_check:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
83 if os.path.isfile(path):
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 944
diff changeset
84 metadata = self._createItemMetadata(path)
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
85 return ContentItem(path, metadata)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
86 return None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
87
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
88 def setupPrepareParser(self, parser, app):
944
23052bf8a62b sources: Change argument name in default source's `createContent`.
Ludovic Chabant <ludovic@chabant.com>
parents: 926
diff changeset
89 parser.add_argument('slug', help='The slug for the new page.')
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
90
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
91 def createContent(self, args):
944
23052bf8a62b sources: Change argument name in default source's `createContent`.
Ludovic Chabant <ludovic@chabant.com>
parents: 926
diff changeset
92 slug = args.get('slug')
23052bf8a62b sources: Change argument name in default source's `createContent`.
Ludovic Chabant <ludovic@chabant.com>
parents: 926
diff changeset
93 if not slug:
23052bf8a62b sources: Change argument name in default source's `createContent`.
Ludovic Chabant <ludovic@chabant.com>
parents: 926
diff changeset
94 slug = '_index'
23052bf8a62b sources: Change argument name in default source's `createContent`.
Ludovic Chabant <ludovic@chabant.com>
parents: 926
diff changeset
95 path = os.path.join(self.fs_endpoint_path, slug)
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
96 _, ext = os.path.splitext(path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
97 if ext == '':
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
98 path = '%s.%s' % (path, self.default_auto_format)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
99
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 944
diff changeset
100 metadata = self._createItemMetadata(path)
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
101 config = metadata.setdefault('config', {})
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
102 config.update({'title': uri_to_title(
1127
4cc020ff2537 prepare: Fix crash when preparing new pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
103 os.path.basename(metadata['route_params']['slug']))})
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
104 return ContentItem(path, metadata)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
105
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
106 def getInteractiveFields(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
107 return [
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
108 InteractiveField('slug', InteractiveField.TYPE_STRING,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
109 'new-page')]