Mercurial > piecrust2

annotate piecrust/sources/default.py @ 852:4850f8c21b6e

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
core: Start of the big refactor for PieCrust 3.0. * Everything is a `ContentSource`, including assets directories. * Most content sources are subclasses of the base file-system source. * A source is processed by a "pipeline", and there are 2 built-in pipelines, one for assets and one for pages. The asset pipeline is vaguely functional, but the page pipeline is completely broken right now. * Rewrite the baking process as just running appropriate pipelines on each content item. This should allow for better parallelization.
author Ludovic Chabant <ludovic@chabant.com>
date Wed, 17 May 2017 00:11:48 -0700
parents 58ebf50235a5
children f070a4fc033c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os.path
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import logging
792
58ebf50235a5 routing: Simplify how routes are defined.
Ludovic Chabant <ludovic@chabant.com>
parents: 577
diff changeset
3 from piecrust.routing import RouteParameter
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
4 from piecrust.sources.base import REL_ASSETS, ContentItem
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
5 from piecrust.sources.fs import FSContentSource
576
0c74a6c4533d sources: Add code to support "interactive" metadata acquisition.
Ludovic Chabant <ludovic@chabant.com>
parents: 520
diff changeset
6 from piecrust.sources.interfaces import (
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
7 IPreparingSource, IInteractiveSource, InteractiveField)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
8 from piecrust.sources.mixins import SimpleAssetsSubDirMixin
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
9 from piecrust.uriutil import uri_to_title
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12 logger = logging.getLogger(__name__)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
15 class DefaultContentSource(FSContentSource,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
16 SimpleAssetsSubDirMixin,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
17 IPreparingSource, IInteractiveSource):
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 SOURCE_NAME = 'default'
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 def __init__(self, app, name, config):
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
21 super().__init__(app, name, config)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
22 self.auto_formats = app.config.get('site/auto_formats')
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 self.default_auto_format = app.config.get('site/default_auto_format')
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
24 self.supported_extensions = list(self.auto_formats)
520
bab91fcef741 bake/serve: Improve support for unicode, add slugification options.
Ludovic Chabant <ludovic@chabant.com>
parents: 366
diff changeset
25
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
26 def _createItemMetadata(self, path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
27 return self._doCreateItemMetadata(path)
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
29 def _finalizeContent(self, parent_group, items, groups):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
30 SimpleAssetsSubDirMixin._onFinalizeContent(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
31 self, parent_group, items, groups)
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
33 def _doCreateItemMetadata(self, path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
34 slug = self._makeSlug(path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
35 metadata = {
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
36 'slug': slug
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
37 }
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
38 _, ext = os.path.splitext(path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
39 if ext:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
40 fmt = self.auto_formats.get(ext.lstrip('.'))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
41 if fmt:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
42 metadata['config'] = {'format': fmt}
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
43 return metadata
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
45 def _makeSlug(self, path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
46 rel_path = os.path.relpath(path, self.fs_endpoint_path)
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 slug, ext = os.path.splitext(rel_path)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 slug = slug.replace('\\', '/')
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 if ext.lstrip('.') not in self.supported_extensions:
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 slug += ext
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 if slug.startswith('./'):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 slug = slug[2:]
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 if slug == '_index':
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 slug = ''
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 return slug
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
57 def getRelatedContents(self, item, relationship):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
58 if relationship == REL_ASSETS:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
59 SimpleAssetsSubDirMixin._getRelatedAssetsContents(self, item)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
60 raise NotImplementedError()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
61
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
62 def getSupportedRouteParameters(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
63 return [
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
64 RouteParameter('slug', RouteParameter.TYPE_PATH)]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
65
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
66 def findContent(self, route_params):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
67 uri_path = route_params.get('slug', '')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
68 if not uri_path:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
69 uri_path = '_index'
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
70 path = os.path.join(self.fs_endpoint_path, uri_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
71 _, ext = os.path.splitext(path)
242
f130365568ff internal: Code reorganization to put less stuff in `sources.base`.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
73 if ext == '':
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
74 paths_to_check = [
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
75 '%s.%s' % (path, e)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
76 for e in self.supported_extensions]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
77 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
78 paths_to_check = [path]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
79 for path in paths_to_check:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
80 if os.path.isfile(path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
81 metadata = self._doCreateItemMetadata(path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
82 return ContentItem(path, metadata)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
83 return None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
84
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
85 def setupPrepareParser(self, parser, app):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
86 parser.add_argument('uri', help='The URI for the new page.')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
87
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
88 def createContent(self, args):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
89 if not hasattr(args, 'uri'):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
90 uri = None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
91 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
92 uri = args.uri
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
93 if not uri:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
94 uri = '_index'
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
95 path = os.path.join(self.fs_endpoint_path, uri)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
96 _, ext = os.path.splitext(path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
97 if ext == '':
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
98 path = '%s.%s' % (path, self.default_auto_format)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
99
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
100 metadata = self._doCreateItemMetadata(path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
101 config = metadata.setdefault('config', {})
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
102 config.update({'title': uri_to_title(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
103 os.path.basename(metadata['slug']))})
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
104 return ContentItem(path, metadata)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
105
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
106 def getInteractiveFields(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
107 return [
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
108 InteractiveField('slug', InteractiveField.TYPE_STRING,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents: 792
diff changeset
109 'new-page')]