piecrust2: piecrust/sources/mixins.py annotate

annotate piecrust/sources/mixins.py @ 852:4850f8c21b6e

core: Start of the big refactor for PieCrust 3.0. * Everything is a `ContentSource`, including assets directories. * Most content sources are subclasses of the base file-system source. * A source is processed by a "pipeline", and there are 2 built-in pipelines, one for assets and one for pages. The asset pipeline is vaguely functional, but the page pipeline is completely broken right now. * Rewrite the baking process as just running appropriate pipelines on each content item. This should allow for better parallelization.

author	Ludovic Chabant <ludovic@chabant.com>
date	Wed, 17 May 2017 00:11:48 -0700
parents	ab5c6a8ae90a
children	f070a4fc033c

rev	line source
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import os.path
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import logging
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	3 from piecrust import osutil
440 32c7c2d219d2 performance: Refactor how data is managed to reduce copying. Ludovic Chabant <ludovic@chabant.com> parents: 411 diff changeset	4 from piecrust.data.paginationdata import PaginationData
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	5 from piecrust.sources.base import ContentItem
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	6 from piecrust.sources.interfaces import IPaginationSource
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	7
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	9 logger = logging.getLogger(__name__)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	10
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	11 assets_suffix = '-assets'
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	13
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	14 class ContentSourceIterator(object):
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	15 def __init__(self, source):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16 self.source = source
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18 # This is to permit recursive traversal of the
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19 # iterator chain. It acts as the end.
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20 self.it = None
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22 def __iter__(self):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	23 return self.source.getAllContentItems()
369 4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 280 diff changeset	24
4b1019bb2533 serve: Giant refactor to change how we handle data when serving pages. Ludovic Chabant <ludovic@chabant.com> parents: 280 diff changeset	25
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26 class DateSortIterator(object):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27 def __init__(self, it, reverse=True):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 self.it = it
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29 self.reverse = reverse
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	30
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31 def __iter__(self):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32 return iter(sorted(self.it,
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 key=lambda x: x.datetime, reverse=self.reverse))
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 class PaginationDataBuilderIterator(object):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37 def __init__(self, it):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	38 self.it = it
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	39
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	40 def __iter__(self):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41 for page in self.it:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	42 if page is not None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	43 yield PaginationData(page)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	44 else:
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	45 yield None
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	46
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	47
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	48 class SimplePaginationSourceMixin(IPaginationSource):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 """ Implements the `IPaginationSource` interface in a standard way that
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50 should fit most page sources.
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51 """
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52 def getItemsPerPage(self):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53 return self.config['items_per_page']
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	55 def getSourceIterator(self):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	56 return ContentSourceIterator(self)
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	57
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58 def getSorterIterator(self, it):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 return DateSortIterator(it)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	60
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	61 def getTailIterator(self, it):
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62 return PaginationDataBuilderIterator(it)
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63
f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	65 class SimpleAssetsSubDirMixin:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	66 def _getRelatedAssetsContents(self, item, relationship):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	67 if not item.metadata.get('__has_assets', False):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	68 return None
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	70 assets = {}
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	71 assets_dir = item.spec + assets_suffix
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	72 for f in osutil.listdir(assets_dir):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	73 fpath = os.path.join(assets_dir, f)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	74 name, _ = os.path.splitext(f)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	75 if name in assets:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	76 raise Exception("Multiple assets are named '%s'." %
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	77 name)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	78 assets[name] = ContentItem(fpath, {'__is_asset': True})
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	79 return assets
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	81 def _onFinalizeContent(self, parent_group, items, groups):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	82 assetsGroups = []
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	83 for g in groups:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	84 if not g.spec.endswith(assets_suffix):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	85 continue
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	86 match = g.spec[:-len(assets_suffix)]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	87 item = next(filter(lambda i: i.spec == match), None)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	88 if item:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	89 item.metadata['__has_assets'] = True
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	90 assetsGroups.append(g)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	91 for g in assetsGroups:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 711 diff changeset	92 groups.remove(g)
242 f130365568ff internal: Code reorganization to put less stuff in `sources.base`. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	93

Mercurial > piecrust2

annotate piecrust/sources/mixins.py @ 852:4850f8c21b6e