piecrust2: piecrust/sources/taxonomy.py annotate

annotate piecrust/sources/taxonomy.py @ 1188:a7c43131d871

bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(

author	Ludovic Chabant <ludovic@chabant.com>
date	Tue, 15 Jun 2021 22:36:23 -0700
parents	9f3e702a8a69
children

rev	line source
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import re
882 acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	2 import copy
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import logging
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import unidecode
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 from piecrust.configuration import ConfigurationError
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 from piecrust.data.filters import (
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	7 PaginationFilter, SettingFilterClause)
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	8 from piecrust.page import Page
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	9 from piecrust.pipelines._pagebaker import PageBaker
1136 5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	10 from piecrust.pipelines._pagerecords import PagePipelineRecordEntry
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	11 from piecrust.pipelines.base import (
1136 5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	12 ContentPipeline, get_record_name_for_source, create_job)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13 from piecrust.routing import RouteParameter
856 9bb22bbe093c refactor: Make the blog archives functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	14 from piecrust.sources.base import ContentItem
9bb22bbe093c refactor: Make the blog archives functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	15 from piecrust.sources.generator import GeneratorSourceBase
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18 logger = logging.getLogger(__name__)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21 SLUGIFY_ENCODE = 1
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22 SLUGIFY_TRANSLITERATE = 2
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	23 SLUGIFY_LOWERCASE = 4
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24 SLUGIFY_DOT_TO_DASH = 8
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	25 SLUGIFY_SPACE_TO_DASH = 16
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 re_first_dot_to_dash = re.compile(r'^\.+')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29 re_dot_to_dash = re.compile(r'\.+')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	30 re_space_to_dash = re.compile(r'\s+')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 class Taxonomy(object):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	34 """ Describes a taxonomy.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	35 """
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 def __init__(self, name, config):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37 self.name = name
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	38 self.config = config
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	39 self.term_name = config.get('term', name)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	40 self.is_multiple = bool(config.get('multiple', False))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41 self.separator = config.get('separator', '/')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42 self.page_ref = config.get('page')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	43
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44 @property
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	45 def setting_name(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	46 if self.is_multiple:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	47 return self.name
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	48 return self.term_name
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	51 _taxonomy_index = """---
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	52 layout: %(template)s
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	53 ---
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	54 """
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	55
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	56
856 9bb22bbe093c refactor: Make the blog archives functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	57 class TaxonomySource(GeneratorSourceBase):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	58 """ A content source that generates taxonomy listing pages.
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 """
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	60 SOURCE_NAME = 'taxonomy'
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	61 DEFAULT_PIPELINE_NAME = 'taxonomy'
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63 def __init__(self, app, name, config):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64 super().__init__(app, name, config)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	65
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	66 tax_name = config.get('taxonomy')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	67 if tax_name is None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 raise ConfigurationError(
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	69 "Taxonomy source '%s' requires a taxonomy name." % name)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	70 self.taxonomy = _get_taxonomy(app, tax_name)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	72 sm = config.get('slugify_mode')
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	73 self.slugifier = _get_slugifier(app, self.taxonomy, sm)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	74
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	75 tpl_name = config.get('template', '_%s.html' % tax_name)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	76 self._raw_item = _taxonomy_index % {'template': tpl_name}
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	77
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	78 def getSupportedRouteParameters(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	79 name = self.taxonomy.term_name
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80 param_type = (RouteParameter.TYPE_PATH if self.taxonomy.is_multiple
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	81 else RouteParameter.TYPE_STRING)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	82 return [RouteParameter(name, param_type,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	83 variadic=self.taxonomy.is_multiple)]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	85 def findContentFromRoute(self, route_params):
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	86 slugified_term = route_params[self.taxonomy.term_name]
1144 9f3e702a8a69 bake: Give unique source specs to each taxonomy or blog archive page. Ludovic Chabant <ludovic@chabant.com> parents: 1136 diff changeset	87 spec = '_index[%s]' % slugified_term
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	88 metadata = {'term': slugified_term,
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	89 'route_params': {
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	90 self.taxonomy.term_name: slugified_term}
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	91 }
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	92 return ContentItem(spec, metadata)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	93
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	94 def slugify(self, term):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	95 return self.slugifier.slugify(term)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	96
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	97 def slugifyMultiple(self, terms):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	98 return self.slugifier.slugifyMultiple(terms)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	99
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	100 def prepareRenderContext(self, ctx):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	101 # Set the pagination source as the source we're generating for.
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	102 ctx.pagination_source = self.inner_source
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	103
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	104 # Get the taxonomy terms from the route metadata... this can come from
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	105 # the browser's URL (while serving) or from the baking (see `bake`
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	106 # method below). In both cases, we expect to have the slugified
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	107 # version of the term, because we're going to set a filter that also
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	108 # slugifies the terms found on each page.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	109 #
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	110 # This is because:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	111 # * while serving, we get everything from the request URL, so we only
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	112 # have the slugified version.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	113 # * if 2 slightly different terms "collide" into the same slugified
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	114 # term, we'll get a merge of the 2 on the listing page, which is
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	115 # what the user expects.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	116 #
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	117 route_params = ctx.page.source_metadata['route_params']
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	118 tax_terms, is_combination = self._getTaxonomyTerms(route_params)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	119 self._setTaxonomyFilter(ctx, tax_terms, is_combination)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	120
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	121 # Add some custom data for rendering.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	122 ctx.custom_data.update({
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	123 self.taxonomy.term_name: tax_terms,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	124 'is_multiple_%s' % self.taxonomy.term_name: is_combination})
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	125 # Add some "plural" version of the term... so for instance, if this
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	126 # is the "tags" taxonomy, "tag" will have one term most of the time,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	127 # except when it's a combination. Here, we add "tags" as something that
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	128 # is always a tuple, even when it's not a combination.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	129 if (self.taxonomy.is_multiple and
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	130 self.taxonomy.name != self.taxonomy.term_name):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	131 mult_val = tax_terms
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	132 if not is_combination:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	133 mult_val = (mult_val,)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	134 ctx.custom_data[self.taxonomy.name] = mult_val
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	135
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	136 def _getTaxonomyTerms(self, route_params):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	137 # Get the individual slugified terms from the route metadata.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	138 all_values = route_params.get(self.taxonomy.term_name)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	139 if all_values is None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	140 raise Exception("'%s' values couldn't be found in route metadata" %
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	141 self.taxonomy.term_name)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	142
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	143 # If it's a "multiple" taxonomy, we need to potentially split the
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	144 # route value into the individual terms (_e.g._ when listing all pages
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	145 # that have 2 given tags, we need to get each of those 2 tags).
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	146 if self.taxonomy.is_multiple:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	147 sep = self.taxonomy.separator
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	148 if sep in all_values:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	149 return tuple(all_values.split(sep)), True
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150 # Not a "multiple" taxonomy, so there's only the one value.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	151 return all_values, False
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	152
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	153 def _setTaxonomyFilter(self, ctx, term_value, is_combination):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	154 # Set up the filter that will check the pages' terms.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	155 flt = PaginationFilter()
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	156 flt.addClause(HasTaxonomyTermsFilterClause(
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	157 self.taxonomy, self.slugifier.mode, term_value, is_combination))
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	158 ctx.pagination_filter = flt
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	159
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	160 def onRouteFunctionUsed(self, route_params):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	161 # Get the values, and slugify them appropriately.
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	162 # If this is a "multiple" taxonomy, `values` will be a tuple of
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	163 # terms. If not, `values` will just be a term.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	164 values = route_params[self.taxonomy.term_name]
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	165 tax_is_multiple = self.taxonomy.is_multiple
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	166 if tax_is_multiple:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	167 slugified_values = self.slugifyMultiple((str(v) for v in values))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	168 route_val = self.taxonomy.separator.join(slugified_values)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	169 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	170 slugified_values = self.slugify(str(values))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	171 route_val = slugified_values
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	172
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	173 # We need to register this use of a taxonomy term.
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	174 # Because the render info gets serialized across bake worker
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	175 # processes, we can only use basic JSON-able structures, which
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	176 # excludes `set`... hence the awkward use of `list`.
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	177 # Also, note that the tuples we're putting in there will be
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	178 # transformed into lists so we'll have to convert back.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	179 rcs = self.app.env.render_ctx_stack
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	180 ri = rcs.current_ctx.render_info
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	181 utt = ri.get('used_taxonomy_terms')
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	182 if utt is None:
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	183 ri['used_taxonomy_terms'] = [slugified_values]
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	184 else:
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	185 if slugified_values not in utt:
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	186 utt.append(slugified_values)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	187
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	188 # Put the slugified values in the route metadata so they're used to
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	189 # generate the URL.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	190 route_params[self.taxonomy.term_name] = route_val
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	191
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	192
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	193 class HasTaxonomyTermsFilterClause(SettingFilterClause):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	194 def __init__(self, taxonomy, slugify_mode, value, is_combination):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	195 super().__init__(taxonomy.setting_name, value)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	196 self._taxonomy = taxonomy
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	197 self._is_combination = is_combination
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	198 self._slugifier = _Slugifier(taxonomy, slugify_mode)
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	199 if taxonomy.is_multiple:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	200 self.pageMatches = self._pageMatchesAny
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	201 else:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	202 self.pageMatches = self._pageMatchesSingle
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	203
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	204 def _pageMatchesAny(self, fil, page):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	205 # Multiple taxonomy, i.e. it supports multiple terms, like tags.
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	206 page_values = page.config.get(self.name)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	207 if page_values is None or not isinstance(page_values, list):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	208 return False
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	209
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	210 page_set = set(map(self._slugifier.slugify, page_values))
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	211 if self._is_combination:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	212 # Multiple taxonomy, and multiple terms to match. Check that
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	213 # the ones to match are all in the page's terms.
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	214 value_set = set(self.value)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	215 return value_set.issubset(page_set)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	216 else:
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	217 # Multiple taxonomy, one term to match.
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	218 return self.value in page_set
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	219
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	220 def _pageMatchesSingle(self, fil, page):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	221 # Single taxonomy. Just compare the values.
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	222 page_value = page.config.get(self.name)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	223 if page_value is None:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	224 return False
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	225 page_value = self._slugifier.slugify(page_value)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	226 return page_value == self.value
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	227
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	228
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	229 def _get_taxonomy(app, tax_name):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	230 tax_config = app.config.get('site/taxonomies/' + tax_name)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	231 if tax_config is None:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	232 raise ConfigurationError("No such taxonomy: %s" % tax_name)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	233 return Taxonomy(tax_name, tax_config)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	234
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	235
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	236 def _get_slugifier(app, taxonomy, slugify_mode=None):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	237 if slugify_mode is None:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	238 slugify_mode = app.config.get('site/slugify_mode', 'encode')
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	239 sm = _parse_slugify_mode(slugify_mode)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	240 return _Slugifier(taxonomy, sm)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	241
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	242
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	243 class TaxonomyPipelineRecordEntry(PagePipelineRecordEntry):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	244 def __init__(self):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	245 super().__init__()
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	246 self.term = None
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	247
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	248
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	249 class TaxonomyPipeline(ContentPipeline):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	250 PIPELINE_NAME = 'taxonomy'
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	251 PASS_NUM = 10
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	252 RECORD_ENTRY_CLASS = TaxonomyPipelineRecordEntry
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	253
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	254 def __init__(self, source, ctx):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	255 if not isinstance(source, TaxonomySource):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	256 raise Exception("The taxonomy pipeline only supports taxonomy "
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	257 "content sources.")
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	258
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	259 super().__init__(source, ctx)
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	260 self.inner_source = source.inner_source
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	261 self.taxonomy = source.taxonomy
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	262 self.slugifier = source.slugifier
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	263 self._tpl_name = source.config['template']
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	264 self._analyzer = None
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	265 self._pagebaker = None
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	266
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	267 def initialize(self):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	268 self._pagebaker = PageBaker(self.app,
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	269 self.ctx.out_dir,
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	270 force=self.ctx.force)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	271 self._pagebaker.startWriterQueue()
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	272
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	273 def shutdown(self):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	274 self._pagebaker.stopWriterQueue()
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	275
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	276 def createJobs(self, ctx):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	277 logger.debug("Building '%s' taxonomy pages for source: %s" %
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	278 (self.taxonomy.name, self.inner_source.name))
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	279 self._analyzer = _TaxonomyTermsAnalyzer(self, ctx.record_histories)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	280 self._analyzer.analyze()
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	281
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	282 logger.debug("Queuing %d '%s' jobs." %
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	283 (len(self._analyzer.dirty_slugified_terms),
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	284 self.taxonomy.name))
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	285 jobs = []
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	286 rec_fac = self.createRecordEntry
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	287 current_record = ctx.current_record
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	288
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	289 for slugified_term in self._analyzer.dirty_slugified_terms:
1144 9f3e702a8a69 bake: Give unique source specs to each taxonomy or blog archive page. Ludovic Chabant <ludovic@chabant.com> parents: 1136 diff changeset	290 item_spec = '_index[%s]' % slugified_term
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	291
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	292 jobs.append(create_job(self, item_spec,
1144 9f3e702a8a69 bake: Give unique source specs to each taxonomy or blog archive page. Ludovic Chabant <ludovic@chabant.com> parents: 1136 diff changeset	293 term=slugified_term))
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	294
1144 9f3e702a8a69 bake: Give unique source specs to each taxonomy or blog archive page. Ludovic Chabant <ludovic@chabant.com> parents: 1136 diff changeset	295 entry = rec_fac(item_spec)
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	296 current_record.addEntry(entry)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	297
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	298 if len(jobs) > 0:
1136 5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	299 return jobs, "taxonomize"
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	300 return None, None
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	301
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	302 def run(self, job, ctx, result):
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	303 term = job['term']
1144 9f3e702a8a69 bake: Give unique source specs to each taxonomy or blog archive page. Ludovic Chabant <ludovic@chabant.com> parents: 1136 diff changeset	304 content_item = ContentItem('_index[%s]' % term,
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	305 {'term': term,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	306 'route_params': {
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	307 self.taxonomy.term_name: term}
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	308 })
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	309 page = Page(self.source, content_item)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	310
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	311 logger.debug("Rendering '%s' page: %s" %
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	312 (self.taxonomy.name, page.source_metadata['term']))
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	313 prev_entry = ctx.previous_entry
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	314 rdr_subs = self._pagebaker.bake(page, prev_entry)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	315
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	316 result['subs'] = rdr_subs
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	317 result['term'] = page.source_metadata['term']
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	318
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	319 def handleJobResult(self, result, ctx):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	320 existing = ctx.record_entry
1136 5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	321 existing.subs = result['subs']
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	322 existing.term = result['term']
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	323
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	324 def postJobRun(self, ctx):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	325 # We create bake entries for all the terms that were not dirty.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	326 # This is because otherwise, on the next incremental bake, we wouldn't
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	327 # find any entry for those things, and figure that we need to delete
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	328 # their outputs.
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	329 analyzer = self._analyzer
882 acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	330 record = ctx.record_history.current
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	331 for prev, cur in ctx.record_history.diffs:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	332 # Only consider entries that don't have any current version
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	333 # (i.e. they weren't baked just now).
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	334 if prev and not cur:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	335 t = prev.term
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	336 if analyzer.isKnownSlugifiedTerm(t):
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	337 logger.debug("Creating unbaked entry for '%s' term: %s" %
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	338 (self.taxonomy.name, t))
882 acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	339 cur = copy.deepcopy(prev)
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	340 cur.flags = \
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	341 PagePipelineRecordEntry.FLAG_COLLAPSED_FROM_LAST_RUN
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	342 record.addEntry(cur)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	343 else:
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	344 logger.debug("Term '%s' in '%s' isn't used anymore." %
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	345 (t, self.taxonomy.name))
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	346
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	347
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	348 class _TaxonomyTermsAnalyzer(object):
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	349 def __init__(self, pipeline, record_histories):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	350 self.pipeline = pipeline
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	351 self.record_histories = record_histories
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	352 self._all_terms = {}
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	353 self._all_dirty_slugified_terms = None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	354
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	355 @property
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	356 def dirty_slugified_terms(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	357 """ Returns the slugified terms that have been 'dirtied' during
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	358 this bake.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	359 """
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	360 return self._all_dirty_slugified_terms
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	361
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	362 def isKnownSlugifiedTerm(self, term):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	363 """ Returns whether the given slugified term has been seen during
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	364 this bake.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	365 """
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	366 return term in self._all_terms
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	367
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	368 def analyze(self):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	369 # Build the list of terms for our taxonomy, and figure out which ones
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	370 # are 'dirty' for the current bake.
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	371 source = self.pipeline.inner_source
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	372 taxonomy = self.pipeline.taxonomy
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	373 slugifier = self.pipeline.slugifier
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	374
1136 5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	375 tax_is_mult = taxonomy.is_multiple
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	376 tax_setting_name = taxonomy.setting_name
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	377
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	378 # First, go over all of our source's pages seen during this bake.
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	379 # Gather all the taxonomy terms they have, and also keep track of
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	380 # the ones used by the pages that were actually rendered (instead of
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	381 # those that were up-to-date and skipped).
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	382 single_dirty_slugified_terms = set()
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	383 current_records = self.record_histories.current
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	384 record_name = get_record_name_for_source(source)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	385 cur_rec = current_records.getRecord(record_name)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	386 for cur_entry in cur_rec.getEntries():
1136 5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	387 if cur_entry.hasFlag(PagePipelineRecordEntry.FLAG_OVERRIDEN):
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	388 continue
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	389
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	390 cur_terms = cur_entry.config.get(tax_setting_name)
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	391 if not cur_terms:
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	392 continue
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	393
1136 5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	394 if not tax_is_mult:
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	395 self._addTerm(
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	396 slugifier, cur_entry.item_spec, cur_terms)
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	397 else:
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	398 self._addTerms(
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	399 slugifier, cur_entry.item_spec, cur_terms)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	400
1136 5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	401 if cur_entry.hasFlag(
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	402 PagePipelineRecordEntry.FLAG_SEGMENTS_RENDERED):
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	403 if not tax_is_mult:
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	404 single_dirty_slugified_terms.add(
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	405 slugifier.slugify(cur_terms))
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	406 else:
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	407 single_dirty_slugified_terms.update(
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	408 (slugifier.slugify(t)
5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	409 for t in cur_terms))
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	410
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	411 self._all_dirty_slugified_terms = list(
1136 5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	412 single_dirty_slugified_terms)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	413 logger.debug("Gathered %d dirty taxonomy terms",
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	414 len(self._all_dirty_slugified_terms))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	415
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	416 # Re-bake the combination pages for terms that are 'dirty'.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	417 # We make all terms into tuple, even those that are not actual
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	418 # combinations, so that we have less things to test further down the
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	419 # line.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	420 #
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	421 # Add the combinations to that list. We get those combinations from
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	422 # wherever combinations were used, so they're coming from the
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	423 # `onRouteFunctionUsed` method. And because combinations can be used
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	424 # by any page in the website (anywhere someone can ask for an URL
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	425 # to the combination page), it means we check all the records, not
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	426 # just the record for our source.
1136 5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	427 if tax_is_mult:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	428 known_combinations = set()
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	429 for rec in current_records.records:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	430 # Cheap way to test if a record contains entries that
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	431 # are sub-types of a page entry: test the first one.
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	432 first_entry = next(iter(rec.getEntries()), None)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	433 if (first_entry is None or
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	434 not isinstance(first_entry, PagePipelineRecordEntry)):
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	435 continue
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	436
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	437 for cur_entry in rec.getEntries():
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	438 used_terms = _get_all_entry_taxonomy_terms(cur_entry)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	439 for terms in used_terms:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	440 if len(terms) > 1:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	441 known_combinations.add(terms)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	442
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	443 dcc = 0
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	444 for terms in known_combinations:
1136 5f97b5b59dfe bake: Optimize cache handling for the baking process. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	445 if not single_dirty_slugified_terms.isdisjoint(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	446 set(terms)):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	447 self._all_dirty_slugified_terms.append(
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	448 taxonomy.separator.join(terms))
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	449 dcc += 1
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	450 logger.debug("Gathered %d term combinations, with %d dirty." %
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	451 (len(known_combinations), dcc))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	452
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	453 def _addTerms(self, slugifier, item_spec, terms):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	454 for t in terms:
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	455 self._addTerm(slugifier, item_spec, t)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	456
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	457 def _addTerm(self, slugifier, item_spec, term):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	458 st = slugifier.slugify(term)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	459 orig_terms = self._all_terms.setdefault(st, [])
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	460 if orig_terms and orig_terms[0] != term:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	461 logger.warning(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	462 "Term '%s' in '%s' is slugified to '%s' which conflicts with "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	463 "previously existing '%s'. The two will be merged." %
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	464 (term, item_spec, st, orig_terms[0]))
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	465 orig_terms.append(term)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	466
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	467
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	468 def _get_all_entry_taxonomy_terms(entry):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	469 res = set()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	470 for o in entry.subs:
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	471 pinfo = o['render_info']
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	472 terms = pinfo.get('used_taxonomy_terms')
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	473 if terms:
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	474 res \|= set([tuple(t) for t in terms])
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	475 return res
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	476
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	477
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	478 class _Slugifier(object):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	479 def __init__(self, taxonomy, mode):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	480 self.taxonomy = taxonomy
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	481 self.mode = mode
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	482
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	483 def slugifyMultiple(self, terms):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	484 return tuple(map(self.slugify, terms))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	485
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	486 def slugify(self, term):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	487 if self.mode & SLUGIFY_TRANSLITERATE:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	488 term = unidecode.unidecode(term)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	489 if self.mode & SLUGIFY_LOWERCASE:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	490 term = term.lower()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	491 if self.mode & SLUGIFY_DOT_TO_DASH:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	492 term = re_first_dot_to_dash.sub('', term)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	493 term = re_dot_to_dash.sub('-', term)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	494 if self.mode & SLUGIFY_SPACE_TO_DASH:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	495 term = re_space_to_dash.sub('-', term)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	496 return term
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	497
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	498
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	499 def _parse_slugify_mode(value):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	500 mapping = {
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	501 'encode': SLUGIFY_ENCODE,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	502 'transliterate': SLUGIFY_TRANSLITERATE,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	503 'lowercase': SLUGIFY_LOWERCASE,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	504 'dot_to_dash': SLUGIFY_DOT_TO_DASH,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	505 'space_to_dash': SLUGIFY_SPACE_TO_DASH}
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	506 mode = 0
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	507 for v in value.split(','):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	508 f = mapping.get(v.strip())
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	509 if f is None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	510 if v == 'iconv':
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	511 raise Exception("'iconv' is not supported as a slugify mode "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	512 "in PieCrust2. Use 'transliterate'.")
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	513 raise Exception("Unknown slugify flag: %s" % v)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	514 mode \|= f
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	515 return mode
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	516

Mercurial > piecrust2

annotate piecrust/sources/taxonomy.py @ 1188:a7c43131d871