piecrust2: piecrust/sources/autoconfig.py annotate

annotate piecrust/sources/autoconfig.py @ 411:e7b865f8f335

bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.

author	Ludovic Chabant <ludovic@chabant.com>
date	Fri, 12 Jun 2015 17:09:19 -0700
parents	dd25bd3ce1f9
children	883a5544cd7f

rev	line source
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	1 import re
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import os.path
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import logging
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	5 from piecrust.configuration import ConfigurationError
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 from piecrust.sources.base import (
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	7 PageSource, PageFactory, InvalidFileSystemEndpointError)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	8 from piecrust.sources.default import (
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	9 filter_page_dirname, filter_page_filename)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	10 from piecrust.sources.interfaces import IListableSource
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	11 from piecrust.sources.mixins import SimplePaginationSourceMixin
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	14 logger = logging.getLogger(__name__)
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	15
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	17 class AutoConfigSourceBase(PageSource, SimplePaginationSourceMixin,
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	18 IListableSource):
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	19 """ Base class for page sources that automatically apply configuration
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	20 settings to their generated pages based on those pages' paths.
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	21 """
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22 def __init__(self, app, name, config):
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	23 super(AutoConfigSourceBase, self).__init__(app, name, config)
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	24 self.fs_endpoint = config.get('fs_endpoint', name)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	25 self.fs_endpoint_path = os.path.join(self.root_dir, self.fs_endpoint)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	26 self.supported_extensions = list(
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	27 app.config.get('site/auto_formats').keys())
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	28 self.default_auto_format = app.config.get('site/default_auto_format')
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	29
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	30 self.capture_mode = config.get('capture_mode', 'path')
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	31 if self.capture_mode not in ['path', 'dirname', 'filename']:
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	32 raise ConfigurationError("Capture mode in source '%s' must be "
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	33 "one of: path, dirname, filename" %
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	34 name)
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 def buildPageFactories(self):
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	37 logger.debug("Scanning for pages in: %s" % self.fs_endpoint_path)
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	38 if not os.path.isdir(self.fs_endpoint_path):
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	39 raise InvalidFileSystemEndpointError(self.name,
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	40 self.fs_endpoint_path)
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42 for dirpath, dirnames, filenames in os.walk(self.fs_endpoint_path):
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	43 rel_dirpath = os.path.relpath(dirpath, self.fs_endpoint_path)
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	44 dirnames[:] = list(filter(filter_page_dirname, dirnames))
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	45
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	46 # If `capture_mode` is `dirname`, we don't need to recompute it
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	47 # for each filename, so we do it here.
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	48 if self.capture_mode == 'dirname':
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	49 config = self._extractConfigFragment(rel_dirpath)
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	50
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	51 for f in filter(filter_page_filename, filenames):
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	52 if self.capture_mode == 'path':
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	53 path = os.path.join(rel_dirpath, f)
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	54 config = self._extractConfigFragment(path)
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	55 elif self.capture_mode == 'filename':
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	56 config = self._extractConfigFragment(f)
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	57
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	58 fac_path = f
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	59 if rel_dirpath != '.':
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	60 fac_path = os.path.join(rel_dirpath, f)
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	61
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	62 slug = self._makeSlug(fac_path)
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	63
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64 metadata = {
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	65 'slug': slug,
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	66 'config': config}
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	67 yield PageFactory(self, fac_path, metadata)
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	68
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	69 def resolveRef(self, ref_path):
363 dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	70 path = os.path.normpath(
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	71 os.path.join(self.fs_endpoint_path, ref_path.lstrip("\\/")))
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	72
363 dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	73 config = None
dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	74 if self.capture_mode == 'dirname':
dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	75 config = self._extractConfigFragment(os.path.dirname(ref_path))
dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	76 elif self.capture_mode == 'path':
dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	77 config = self._extractConfigFragment(ref_path)
dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	78 elif self.capture_mode == 'filename':
dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	79 config = self._extractConfigFragment(os.path.basename(ref_path))
dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	80
dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	81 slug = self._makeSlug(ref_path)
dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	82 metadata = {'slug': slug, 'config': config}
dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	83 return path, metadata
dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	84
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	85 def listPath(self, rel_path):
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	86 raise NotImplementedError()
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	87
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	88 def getDirpath(self, rel_path):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	89 return os.path.dirname(rel_path)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	90
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	91 def getBasename(self, rel_path):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	92 filename = os.path.basename(rel_path)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	93 name, _ = os.path.splitext(filename)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	94 return name
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	95
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	96 def _makeSlug(self, rel_path):
267 f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	97 slug = rel_path.replace('\\', '/')
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	98 slug = self._cleanSlug(slug)
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	99 slug, ext = os.path.splitext(slug)
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	100 if ext.lstrip('.') not in self.supported_extensions:
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	101 slug += ext
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	102 if slug.startswith('./'):
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	103 slug = slug[2:]
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	104 if slug == '_index':
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	105 slug = ''
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	106 return slug
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	107
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	108 def _cleanSlug(self, slug):
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	109 return slug
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	110
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	111 def _extractConfigFragment(self, rel_path):
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	112 raise NotImplementedError()
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	113
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	114
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	115 class AutoConfigSource(AutoConfigSourceBase):
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	116 """ Page source that extracts configuration settings from the sub-folders
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	117 each page resides in. This is ideal for setting tags or categories
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	118 on pages based on the folders they're in.
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	119 """
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	120 SOURCE_NAME = 'autoconfig'
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	121
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	122 def __init__(self, app, name, config):
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	123 config['capture_mode'] = 'dirname'
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	124 super(AutoConfigSource, self).__init__(app, name, config)
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	125 self.setting_name = config.get('setting_name', name)
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	126 self.only_single_values = config.get('only_single_values', False)
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	127 self.collapse_single_values = config.get('collapse_single_values',
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	128 False)
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	129 self.supported_extensions = list(
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	130 app.config.get('site/auto_formats').keys())
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	131
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	132 def _extractConfigFragment(self, rel_path):
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	133 if rel_path == '.':
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	134 values = []
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	135 else:
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	136 values = rel_path.split(os.sep)
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	137
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	138 if self.only_single_values:
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	139 if len(values) > 1:
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	140 raise Exception("Only one folder level is allowed for pages "
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	141 "in source '%s'." % self.name)
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	142 elif len(values) == 1:
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	143 values = values[0]
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	144 else:
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	145 values = None
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	146
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	147 if self.collapse_single_values:
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	148 if len(values) == 1:
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	149 values = values[0]
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	150 elif len(values) == 0:
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	151 values = None
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	152
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	153 return {self.setting_name: values}
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	154
363 dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	155 def findPageFactory(self, metadata, mode):
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	156 # Pages from this source are effectively flattened, so we need to
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	157 # find pages using a brute-force kinda way.
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	158 for dirpath, dirnames, filenames in os.walk(self.fs_endpoint_path):
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	159 for f in filenames:
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	160 slug, _ = os.path.splitext(f)
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	161 if slug == metadata['slug']:
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	162 path = os.path.join(dirpath, f)
432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	163 rel_path = os.path.relpath(path, self.fs_endpoint_path)
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	164 config = self._extractConfigFragment(rel_path)
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	165 metadata = {'slug': slug, 'config': config}
363 dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	166 return PageFactory(self, rel_path, metadata)
dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	167 return None
148 432cd534ce08 Add `autoconfig` page source. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	168
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	169 def listPath(self, rel_path):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	170 rel_path = rel_path.lstrip('\\/')
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	171 path = os.path.join(self.fs_endpoint_path, rel_path)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	172 names = sorted(os.listdir(path))
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	173 items = []
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	174 for name in names:
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	175 if os.path.isdir(os.path.join(path, name)):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	176 if filter_page_dirname(name):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	177 rel_subdir = os.path.join(rel_path, name)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	178 items.append((True, name, rel_subdir))
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	179 else:
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	180 if filter_page_filename(name):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	181 cur_rel_path = os.path.join(rel_path, name)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	182 slug = self._makeSlug(cur_rel_path)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	183 config = self._extractConfigFragment(cur_rel_path)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	184 metadata = {'slug': slug, 'config': config}
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	185 fac = PageFactory(self, cur_rel_path, metadata)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	186
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	187 name, _ = os.path.splitext(name)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	188 items.append((False, name, fac))
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	189 return items
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	190
267 f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	191 def _cleanSlug(self, slug):
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	192 return os.path.basename(slug)
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	193
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	194
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	195 class OrderedPageSource(AutoConfigSourceBase):
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	196 """ A page source that assigns an "order" to its pages based on a
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	197 numerical prefix in their filename. Page iterators will automatically
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	198 sort pages using that order.
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	199 """
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	200 SOURCE_NAME = 'ordered'
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	201
286 a2d283d1033d tests: Fixes for running on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 267 diff changeset	202 re_pattern = re.compile(r'(^\|[/\\])(?P<num>\d+)_')
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	203
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	204 def __init__(self, app, name, config):
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	205 config['capture_mode'] = 'path'
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	206 super(OrderedPageSource, self).__init__(app, name, config)
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	207 self.setting_name = config.get('setting_name', 'order')
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	208 self.default_value = config.get('default_value', 0)
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	209 self.supported_extensions = list(
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	210 app.config.get('site/auto_formats').keys())
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	211
363 dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	212 def findPageFactory(self, metadata, mode):
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	213 uri_path = metadata.get('slug', '')
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	214 if uri_path == '':
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	215 uri_path = '_index'
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	216
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	217 path = self.fs_endpoint_path
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	218 uri_parts = uri_path.split('/')
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	219 for i, p in enumerate(uri_parts):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	220 if i == len(uri_parts) - 1:
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	221 # Last part, this is the filename. We need to check for either
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	222 # the name, or the name with the prefix, but also handle a
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	223 # possible extension.
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	224 p_pat = r'(\d+_)?' + re.escape(p)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	225
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	226 _, ext = os.path.splitext(uri_path)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	227 if ext == '':
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	228 p_pat += r'\.[\w\d]+'
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	229
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	230 found = False
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	231 for name in os.listdir(path):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	232 if re.match(p_pat, name):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	233 path = os.path.join(path, name)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	234 found = True
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	235 break
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	236 if not found:
363 dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	237 return None
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	238 else:
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	239 # Find each sub-directory. It can either be a directory with
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	240 # the name itself, or the name with a number prefix.
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	241 p_pat = r'(\d+_)?' + re.escape(p)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	242 found = False
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	243 for name in os.listdir(path):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	244 if re.match(p_pat, name):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	245 path = os.path.join(path, name)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	246 found = True
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	247 break
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	248 if not found:
363 dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	249 return None
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	250
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	251 fac_path = os.path.relpath(path, self.fs_endpoint_path)
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	252 config = self._extractConfigFragment(fac_path)
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	253 metadata = {'slug': uri_path, 'config': config}
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	254
363 dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 360 diff changeset	255 return PageFactory(self, fac_path, metadata)
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	256
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	257 def getSorterIterator(self, it):
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	258 accessor = self.getSettingAccessor()
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	259 return OrderTrailSortIterator(it, self.setting_name + '_trail',
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	260 value_accessor=accessor)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	261
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	262 def listPath(self, rel_path):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	263 rel_path = rel_path.lstrip('/')
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	264 path = self.fs_endpoint_path
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	265 if rel_path != '':
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	266 parts = rel_path.split('/')
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	267 for p in parts:
360 95874699ec2e sources: Fix how the `autoconfig` source iterates over its structure. Ludovic Chabant <ludovic@chabant.com> parents: 286 diff changeset	268 p_pat = r'(\d+_)?' + re.escape(p) + '$'
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	269 for name in os.listdir(path):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	270 if re.match(p_pat, name):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	271 path = os.path.join(path, name)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	272 break
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	273 else:
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	274 raise Exception("No such path: %s" % rel_path)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	275
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	276 items = []
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	277 names = sorted(os.listdir(path))
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	278 for name in names:
248 3f740928043a sources: The ordered source returns names without prefixes in `listPath`. Ludovic Chabant <ludovic@chabant.com> parents: 242 diff changeset	279 clean_name = self.re_pattern.sub('', name)
3f740928043a sources: The ordered source returns names without prefixes in `listPath`. Ludovic Chabant <ludovic@chabant.com> parents: 242 diff changeset	280 clean_name, _ = os.path.splitext(clean_name)
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	281 if os.path.isdir(os.path.join(path, name)):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	282 if filter_page_dirname(name):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	283 rel_subdir = os.path.join(rel_path, name)
248 3f740928043a sources: The ordered source returns names without prefixes in `listPath`. Ludovic Chabant <ludovic@chabant.com> parents: 242 diff changeset	284 items.append((True, clean_name, rel_subdir))
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	285 else:
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	286 if filter_page_filename(name):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	287 slug = self._makeSlug(os.path.join(rel_path, name))
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	288
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	289 fac_path = name
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	290 if rel_path != '.':
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	291 fac_path = os.path.join(rel_path, name)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	292 fac_path = fac_path.replace('\\', '/')
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	293
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	294 config = self._extractConfigFragment(fac_path)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	295 metadata = {'slug': slug, 'config': config}
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	296 fac = PageFactory(self, fac_path, metadata)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	297
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	298 name, _ = os.path.splitext(name)
248 3f740928043a sources: The ordered source returns names without prefixes in `listPath`. Ludovic Chabant <ludovic@chabant.com> parents: 242 diff changeset	299 items.append((False, clean_name, fac))
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	300 return items
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	301
267 f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	302 def _cleanSlug(self, slug):
f512905ae812 sources: Generate proper slugs in the `autoconfig` and `ordered` sources. Ludovic Chabant <ludovic@chabant.com> parents: 248 diff changeset	303 return self.re_pattern.sub(r'\1', slug)
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	304
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	305 def _extractConfigFragment(self, rel_path):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	306 values = []
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	307 for m in self.re_pattern.finditer(rel_path):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	308 val = int(m.group('num'))
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	309 values.append(val)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	310
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	311 if len(values) == 0:
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	312 values.append(self.default_value)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	313
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	314 return {
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	315 self.setting_name: values[-1],
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	316 self.setting_name + '_trail': values}
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	317
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	318 def _populateMetadata(self, rel_path, metadata, mode=None):
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	319 _, filename = os.path.split(rel_path)
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	320 config = self._extractConfigFragment(filename)
239 f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	321 metadata['config'] = config
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	322 slug = metadata['slug']
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	323 metadata['slug'] = self.re_pattern.sub(r'\1', slug)
f43f19975671 sources: Refactor `autoconfig` source, add `OrderedPageSource`. Ludovic Chabant <ludovic@chabant.com> parents: 156 diff changeset	324
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	325
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	326 class OrderTrailSortIterator(object):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	327 def __init__(self, it, trail_name, value_accessor):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	328 self.it = it
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	329 self.trail_name = trail_name
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	330 self.value_accessor = value_accessor
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	331
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	332 def __iter__(self):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	333 return iter(sorted(self.it, key=self._key_getter))
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	334
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	335 def _key_getter(self, item):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	336 values = self.value_accessor(item, self.trail_name)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	337 key = ''.join(values)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	338 return key
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: 239 diff changeset	339

Mercurial > piecrust2

annotate piecrust/sources/autoconfig.py @ 411:e7b865f8f335