Mercurial > piecrust2

annotate piecrust/sources/default.py @ 853:f070a4fc033c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
core: Continue PieCrust3 refactor, simplify pages. The asset pipeline is still the only function pipeline at this point. * No more `QualifiedPage`, and several other pieces of code deleted. * Data providers are simpler and more focused. For instance, the page iterator doesn't try to support other types of items. * Route parameters are proper known source metadata to remove the confusion between the two. * Make the baker and pipeline more correctly manage records and record histories. * Add support for record collapsing and deleting stale outputs in the asset pipeline.
author Ludovic Chabant <ludovic@chabant.com>
date Sun, 21 May 2017 00:06:59 -0700
parents 4850f8c21b6e
children 08e02c2a2a1a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os.path
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import logging
792
58ebf50235a5 routing: Simplify how routes are defined.
Ludovic Chabant <ludovic@chabant.com>
parents: 577
diff changeset
3 from piecrust.routing import RouteParameter
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
4 from piecrust.sources.base import REL_ASSETS, ContentItem
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
5 from piecrust.sources.fs import FSContentSource
576
0c74a6c4533d sources: Add code to support "interactive" metadata acquisition.
Ludovic Chabant <ludovic@chabant.com>
parents: 520
diff changeset
6 from piecrust.sources.interfaces import (
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
7 IPreparingSource, IInteractiveSource, InteractiveField)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
8 from piecrust.sources.mixins import SimpleAssetsSubDirMixin
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
9 from piecrust.uriutil import uri_to_title
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12 logger = logging.getLogger(__name__)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
15 class DefaultContentSource(FSContentSource,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
16 SimpleAssetsSubDirMixin,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
17 IPreparingSource, IInteractiveSource):
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 SOURCE_NAME = 'default'
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 def __init__(self, app, name, config):
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
21 super().__init__(app, name, config)
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
22
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
23 config.setdefault('data_type', 'page_iterator')
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
24
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
25 self.auto_formats = app.config.get('site/auto_formats')
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 self.default_auto_format = app.config.get('site/default_auto_format')
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
27 self.supported_extensions = list(self.auto_formats)
520
bab91fcef741 bake/serve: Improve support for unicode, add slugification options.
Ludovic Chabant <ludovic@chabant.com>
parents: 366
diff changeset
28
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
29 def _createItemMetadata(self, path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
30 return self._doCreateItemMetadata(path)
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
32 def _finalizeContent(self, parent_group, items, groups):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
33 SimpleAssetsSubDirMixin._onFinalizeContent(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
34 self, parent_group, items, groups)
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
36 def _doCreateItemMetadata(self, path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
37 slug = self._makeSlug(path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
38 metadata = {
853
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
39 'route_params': {
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
40 'slug': slug
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 852
diff changeset
41 }
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
42 }
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
43 _, ext = os.path.splitext(path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
44 if ext:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
45 fmt = self.auto_formats.get(ext.lstrip('.'))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
46 if fmt:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
47 metadata['config'] = {'format': fmt}
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
48 return metadata
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
50 def _makeSlug(self, path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
51 rel_path = os.path.relpath(path, self.fs_endpoint_path)
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 slug, ext = os.path.splitext(rel_path)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 slug = slug.replace('\\', '/')
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 if ext.lstrip('.') not in self.supported_extensions:
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 slug += ext
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56 if slug.startswith('./'):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 slug = slug[2:]
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 if slug == '_index':
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 slug = ''
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60 return slug
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
62 def getRelatedContents(self, item, relationship):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
63 if relationship == REL_ASSETS:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
64 SimpleAssetsSubDirMixin._getRelatedAssetsContents(self, item)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
65 raise NotImplementedError()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
66
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
67 def getSupportedRouteParameters(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
68 return [
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
69 RouteParameter('slug', RouteParameter.TYPE_PATH)]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
70
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
71 def findContent(self, route_params):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
72 uri_path = route_params.get('slug', '')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
73 if not uri_path:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
74 uri_path = '_index'
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
75 path = os.path.join(self.fs_endpoint_path, uri_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
76 _, ext = os.path.splitext(path)
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
78 if ext == '':
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
79 paths_to_check = [
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
80 '%s.%s' % (path, e)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
81 for e in self.supported_extensions]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
82 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
83 paths_to_check = [path]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
84 for path in paths_to_check:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
85 if os.path.isfile(path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
86 metadata = self._doCreateItemMetadata(path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
87 return ContentItem(path, metadata)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
88 return None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
89
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
90 def setupPrepareParser(self, parser, app):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
91 parser.add_argument('uri', help='The URI for the new page.')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
92
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
93 def createContent(self, args):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
94 if not hasattr(args, 'uri'):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
95 uri = None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
96 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
97 uri = args.uri
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
98 if not uri:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
99 uri = '_index'
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
100 path = os.path.join(self.fs_endpoint_path, uri)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
101 _, ext = os.path.splitext(path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
102 if ext == '':
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
103 path = '%s.%s' % (path, self.default_auto_format)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
104
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
105 metadata = self._doCreateItemMetadata(path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
106 config = metadata.setdefault('config', {})
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
107 config.update({'title': uri_to_title(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
108 os.path.basename(metadata['slug']))})
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
109 return ContentItem(path, metadata)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
110
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
111 def getInteractiveFields(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
112 return [
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
113 InteractiveField('slug', InteractiveField.TYPE_STRING,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
114 'new-page')]