piecrust2: piecrust/sources/fs.py annotate

annotate piecrust/sources/fs.py @ 1188:a7c43131d871

bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(

author	Ludovic Chabant <ludovic@chabant.com>
date	Tue, 15 Jun 2021 22:36:23 -0700
parents	4ea64255eadf
children

rev	line source
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import os.path
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	2 import re
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	3 import glob
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	4 import fnmatch
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import logging
1156 4ea64255eadf sources: Add ability to force a realm on a source. Ludovic Chabant <ludovic@chabant.com> parents: 1078 diff changeset	6 from werkzeug.utils import cached_property
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	7 from piecrust import osutil
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8 from piecrust.routing import RouteParameter
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	9 from piecrust.sources.base import (
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	10 ContentItem, ContentGroup, ContentSource,
1078 a6618fdab37e sources: Fix some invalid relationship name. Ludovic Chabant <ludovic@chabant.com> parents: 1060 diff changeset	11 REL_PARENT_GROUP, REL_LOGICAL_PARENT_ITEM, REL_LOGICAL_CHILD_GROUP)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	14 logger = logging.getLogger(__name__)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	15
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17 class InvalidFileSystemEndpointError(Exception):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18 def __init__(self, source_name, fs_endpoint):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19 super(InvalidFileSystemEndpointError, self).__init__(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20 "Invalid file-system endpoint for source '%s': %s" %
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21 (source_name, fs_endpoint))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	23
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24 def _filter_crap_files(f):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	25 return (f[-1] != '~' and # Vim temp files and what-not
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26 f not in ['.DS_Store', 'Thumbs.db']) # OSX and Windows bullshit
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29 class FSContentSourceBase(ContentSource):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	30 """ Implements some basic stuff for a `ContentSource` that stores its
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31 items as files on disk.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32 """
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 def __init__(self, app, name, config):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34 super().__init__(app, name, config)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35 self.fs_endpoint = config.get('fs_endpoint', name)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 self.fs_endpoint_path = os.path.join(self.root_dir, self.fs_endpoint)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37
1156 4ea64255eadf sources: Add ability to force a realm on a source. Ludovic Chabant <ludovic@chabant.com> parents: 1078 diff changeset	38 @cached_property
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	39 def root_dir(self):
1156 4ea64255eadf sources: Add ability to force a realm on a source. Ludovic Chabant <ludovic@chabant.com> parents: 1078 diff changeset	40 if self.is_theme_source and not self.config.get('force_user_realm'):
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	41 return self.app.theme_dir
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	42 return self.app.root_dir
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	43
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44 def _checkFSEndpoint(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	45 if not os.path.isdir(self.fs_endpoint_path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	46 if self.config.get('ignore_missing_dir'):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	47 return False
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	48 raise InvalidFileSystemEndpointError(self.name,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 self.fs_endpoint_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50 return True
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51
933 7e7fc7926307 sources: File-system sources accept all `open` arguments. Ludovic Chabant <ludovic@chabant.com> parents: 866 diff changeset	52 def openItem(self, item, mode='r', **kwargs):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53 for m in 'wxa+':
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54 if m in mode:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	55 # If opening the file for writing, let's make sure the
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	56 # directory exists.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	57 dirname = os.path.dirname(item.spec)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58 if not os.path.exists(dirname):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 os.makedirs(dirname, 0o755)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	60 break
933 7e7fc7926307 sources: File-system sources accept all `open` arguments. Ludovic Chabant <ludovic@chabant.com> parents: 866 diff changeset	61 return open(item.spec, mode, **kwargs)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63 def getItemMtime(self, item):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64 return os.path.getmtime(item.spec)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	65
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	66 def describe(self):
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	67 return {'endpoint_path': self.fs_endpoint_path}
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	68
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	70 class FSContentSource(FSContentSourceBase):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71 """ Implements a `ContentSource` that simply returns files on disk
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	72 under a given root directory.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	73 """
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	74 SOURCE_NAME = 'fs'
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	76 def __init__(self, app, name, config):
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	77 super().__init__(app, name, config)
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	78
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	79 config.setdefault('data_type', 'asset_iterator')
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	80
1042 895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	81 self._ignore = _parse_patterns(config.get('ignore'))
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	82 self._filter = _parse_patterns(config.get('filter'))
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	83
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84 def getContents(self, group):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	85 if not self._checkFSEndpoint():
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	86 return None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	87
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	88 parent_path = self.fs_endpoint_path
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	89 if group is not None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	90 parent_path = group.spec
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	91
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	92 names = filter(_filter_crap_files, osutil.listdir(parent_path))
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	93
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	94 items = []
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	95 groups = []
1042 895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	96 for name in names:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	97 path = os.path.join(parent_path, name)
1042 895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	98 if self._filterPath(path):
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	99 if os.path.isdir(path):
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	100 metadata = self._createGroupMetadata(path)
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	101 groups.append(ContentGroup(path, metadata))
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	102 else:
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	103 metadata = self._createItemMetadata(path)
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	104 items.append(ContentItem(path, metadata))
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	105 self._finalizeContent(group, items, groups)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	106 return items + groups
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	107
1042 895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	108 def _filterPath(self, path):
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	109 rel_path = os.path.relpath(path, self.fs_endpoint_path)
1042 895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	110
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	111 if self._ignore is not None:
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	112 if _matches_patterns(self._ignore, rel_path):
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	113 return False
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	114
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	115 if self._filter is not None:
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	116 if _matches_patterns(self._filter, rel_path):
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	117 return True
1042 895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	118 return False
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	119
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	120 return True
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	121
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	122 def _createGroupMetadata(self, path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	123 return {}
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	124
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	125 def _createItemMetadata(self, path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	126 return {}
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	127
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	128 def _finalizeContent(self, parent_group, items, groups):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	129 pass
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	130
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	131 def findContentFromSpec(self, spec):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	132 if os.path.isdir(spec):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	133 metadata = self._createGroupMetadata(spec)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	134 return ContentGroup(spec, metadata)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	135 elif os.path.isfile(spec):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	136 metadata = self._createItemMetadata(spec)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	137 return ContentItem(spec, metadata)
866 d9059257743c refactor: Make the linker work again. Ludovic Chabant <ludovic@chabant.com> parents: 862 diff changeset	138 return None
d9059257743c refactor: Make the linker work again. Ludovic Chabant <ludovic@chabant.com> parents: 862 diff changeset	139
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	140 def getRelatedContents(self, item, relationship):
977 84fc72a17f7a sources: Changes in related contents management. Ludovic Chabant <ludovic@chabant.com> parents: 933 diff changeset	141 if relationship == REL_PARENT_GROUP:
84fc72a17f7a sources: Changes in related contents management. Ludovic Chabant <ludovic@chabant.com> parents: 933 diff changeset	142 parent_dir = os.path.dirname(item.spec)
84fc72a17f7a sources: Changes in related contents management. Ludovic Chabant <ludovic@chabant.com> parents: 933 diff changeset	143 if len(parent_dir) >= len(self.fs_endpoint_path):
84fc72a17f7a sources: Changes in related contents management. Ludovic Chabant <ludovic@chabant.com> parents: 933 diff changeset	144 metadata = self._createGroupMetadata(parent_dir)
84fc72a17f7a sources: Changes in related contents management. Ludovic Chabant <ludovic@chabant.com> parents: 933 diff changeset	145 return ContentGroup(parent_dir, metadata)
84fc72a17f7a sources: Changes in related contents management. Ludovic Chabant <ludovic@chabant.com> parents: 933 diff changeset	146
84fc72a17f7a sources: Changes in related contents management. Ludovic Chabant <ludovic@chabant.com> parents: 933 diff changeset	147 # Don't return a group for paths that are outside of our
84fc72a17f7a sources: Changes in related contents management. Ludovic Chabant <ludovic@chabant.com> parents: 933 diff changeset	148 # endpoint directory.
84fc72a17f7a sources: Changes in related contents management. Ludovic Chabant <ludovic@chabant.com> parents: 933 diff changeset	149 return None
84fc72a17f7a sources: Changes in related contents management. Ludovic Chabant <ludovic@chabant.com> parents: 933 diff changeset	150
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	151 if relationship == REL_LOGICAL_PARENT_ITEM:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	152 # If we want the logical parent item of a folder, we find a
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	153 # page file with the same name as the folder.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	154 if not item.is_group:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	155 raise ValueError()
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 977 diff changeset	156 parent_glob = item.spec.rstrip('/\\') + '.*'
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	157 for n in glob.iglob(parent_glob):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	158 if os.path.isfile(n):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	159 metadata = self._createItemMetadata(n)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	160 return ContentItem(n, metadata)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	161 return None
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	162
1078 a6618fdab37e sources: Fix some invalid relationship name. Ludovic Chabant <ludovic@chabant.com> parents: 1060 diff changeset	163 if relationship == REL_LOGICAL_CHILD_GROUP:
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	164 # If we want the children items of an item, we look for
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	165 # a directory that has the same name as the item's file.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	166 if item.is_group:
1060 3678cddf99f9 sources: Improve error message when trying to get children pages on a group. Ludovic Chabant <ludovic@chabant.com> parents: 1042 diff changeset	167 raise ValueError(
3678cddf99f9 sources: Improve error message when trying to get children pages on a group. Ludovic Chabant <ludovic@chabant.com> parents: 1042 diff changeset	168 "'%s' is a content group and doesn't have a logical "
3678cddf99f9 sources: Improve error message when trying to get children pages on a group. Ludovic Chabant <ludovic@chabant.com> parents: 1042 diff changeset	169 "child. Did you call `family.children` on a group? "
3678cddf99f9 sources: Improve error message when trying to get children pages on a group. Ludovic Chabant <ludovic@chabant.com> parents: 1042 diff changeset	170 "You need to check `is_group` first.")
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	171 dir_path, _ = os.path.splitext(item.spec)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	172 if os.path.isdir(dir_path):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	173 metadata = self._createGroupMetadata(dir_path)
977 84fc72a17f7a sources: Changes in related contents management. Ludovic Chabant <ludovic@chabant.com> parents: 933 diff changeset	174 return ContentGroup(dir_path, metadata)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	175 return None
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	176
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	177 return None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	178
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	179 def getSupportedRouteParameters(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	180 return [
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	181 RouteParameter('path', RouteParameter.TYPE_PATH)]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	182
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	183
1042 895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	184 def _parse_patterns(patterns):
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	185 if not patterns:
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	186 return None
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	187
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	188 globs = []
f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	189 regexes = []
1042 895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	190 for pat in patterns:
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	191 if len(pat) > 2 and pat[0] == '/' and pat[-1] == '/':
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	192 regexes.append(re.compile(pat[1:-1]))
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	193 else:
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	194 globs.append(pat)
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	195 return globs, regexes
1042 895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	196
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	197
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	198 def _matches_patterns(patterns, subj):
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	199 globs, regexes = patterns
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	200 if globs:
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	201 for g in globs:
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	202 if fnmatch.fnmatch(subj, g):
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	203 return True
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	204 if regexes:
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	205 for r in regexes:
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	206 if r.search(subj):
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	207 return True
895f49c9833d sources: Adds support for whitelist filtering to file-system sources. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	208 return False

Mercurial > piecrust2

annotate piecrust/sources/fs.py @ 1188:a7c43131d871