piecrust2: piecrust/sources/taxonomy.py annotate

annotate piecrust/sources/taxonomy.py @ 1114:8af2ea1f5c34

tasks: Add new `tasks` command and infrastructure, with `mention` task. * The new command lets `chef` run tasks from a queue. * The webmention endpoint now adds a mention task. * Moved mention handling code to a task runner.

author	Ludovic Chabant <ludovic@chabant.com>
date	Thu, 22 Feb 2018 22:12:45 -0800
parents	1857dbd4580f
children	5f97b5b59dfe

rev	line source
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import re
882 acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	2 import copy
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import logging
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import unidecode
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 from piecrust.configuration import ConfigurationError
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 from piecrust.data.filters import (
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	7 PaginationFilter, SettingFilterClause)
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	8 from piecrust.page import Page
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	9 from piecrust.pipelines._pagebaker import PageBaker
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	10 from piecrust.pipelines._pagerecords import (
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	11 PagePipelineRecordEntry,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	12 add_page_job_result, merge_job_result_into_record_entry)
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	13 from piecrust.pipelines.base import (
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	14 ContentPipeline, get_record_name_for_source,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	15 create_job, content_item_from_job)
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	16 from piecrust.pipelines.records import RecordHistory
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17 from piecrust.routing import RouteParameter
856 9bb22bbe093c refactor: Make the blog archives functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	18 from piecrust.sources.base import ContentItem
9bb22bbe093c refactor: Make the blog archives functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	19 from piecrust.sources.generator import GeneratorSourceBase
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22 logger = logging.getLogger(__name__)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	23
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	25 SLUGIFY_ENCODE = 1
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26 SLUGIFY_TRANSLITERATE = 2
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27 SLUGIFY_LOWERCASE = 4
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 SLUGIFY_DOT_TO_DASH = 8
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29 SLUGIFY_SPACE_TO_DASH = 16
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	30
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32 re_first_dot_to_dash = re.compile(r'^\.+')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 re_dot_to_dash = re.compile(r'\.+')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34 re_space_to_dash = re.compile(r'\s+')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37 class Taxonomy(object):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	38 """ Describes a taxonomy.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	39 """
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	40 def __init__(self, name, config):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41 self.name = name
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42 self.config = config
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	43 self.term_name = config.get('term', name)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44 self.is_multiple = bool(config.get('multiple', False))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	45 self.separator = config.get('separator', '/')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	46 self.page_ref = config.get('page')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	47
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	48 @property
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 def setting_name(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50 if self.is_multiple:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51 return self.name
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52 return self.term_name
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	55 _taxonomy_index = """---
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	56 layout: %(template)s
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	57 ---
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	58 """
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	59
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	60
856 9bb22bbe093c refactor: Make the blog archives functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	61 class TaxonomySource(GeneratorSourceBase):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	62 """ A content source that generates taxonomy listing pages.
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63 """
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64 SOURCE_NAME = 'taxonomy'
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	65 DEFAULT_PIPELINE_NAME = 'taxonomy'
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	66
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	67 def __init__(self, app, name, config):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 super().__init__(app, name, config)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	70 tax_name = config.get('taxonomy')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71 if tax_name is None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	72 raise ConfigurationError(
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	73 "Taxonomy source '%s' requires a taxonomy name." % name)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	74 self.taxonomy = _get_taxonomy(app, tax_name)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76 sm = config.get('slugify_mode')
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	77 self.slugifier = _get_slugifier(app, self.taxonomy, sm)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	78
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	79 tpl_name = config.get('template', '_%s.html' % tax_name)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	80 self._raw_item = _taxonomy_index % {'template': tpl_name}
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	81
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	82 def getSupportedRouteParameters(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	83 name = self.taxonomy.term_name
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84 param_type = (RouteParameter.TYPE_PATH if self.taxonomy.is_multiple
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	85 else RouteParameter.TYPE_STRING)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	86 return [RouteParameter(name, param_type,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	87 variadic=self.taxonomy.is_multiple)]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	88
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	89 def findContentFromRoute(self, route_params):
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	90 slugified_term = route_params[self.taxonomy.term_name]
876 d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	91 spec = '_index'
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	92 metadata = {'term': slugified_term,
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	93 'route_params': {
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	94 self.taxonomy.term_name: slugified_term}
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	95 }
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	96 return ContentItem(spec, metadata)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	97
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	98 def slugify(self, term):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	99 return self.slugifier.slugify(term)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	100
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	101 def slugifyMultiple(self, terms):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	102 return self.slugifier.slugifyMultiple(terms)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	103
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	104 def prepareRenderContext(self, ctx):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	105 # Set the pagination source as the source we're generating for.
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	106 ctx.pagination_source = self.inner_source
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	107
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	108 # Get the taxonomy terms from the route metadata... this can come from
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	109 # the browser's URL (while serving) or from the baking (see `bake`
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	110 # method below). In both cases, we expect to have the slugified
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	111 # version of the term, because we're going to set a filter that also
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	112 # slugifies the terms found on each page.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	113 #
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	114 # This is because:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	115 # * while serving, we get everything from the request URL, so we only
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	116 # have the slugified version.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	117 # * if 2 slightly different terms "collide" into the same slugified
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	118 # term, we'll get a merge of the 2 on the listing page, which is
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	119 # what the user expects.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	120 #
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	121 route_params = ctx.page.source_metadata['route_params']
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	122 tax_terms, is_combination = self._getTaxonomyTerms(route_params)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	123 self._setTaxonomyFilter(ctx, tax_terms, is_combination)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	124
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	125 # Add some custom data for rendering.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	126 ctx.custom_data.update({
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	127 self.taxonomy.term_name: tax_terms,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	128 'is_multiple_%s' % self.taxonomy.term_name: is_combination})
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	129 # Add some "plural" version of the term... so for instance, if this
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	130 # is the "tags" taxonomy, "tag" will have one term most of the time,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	131 # except when it's a combination. Here, we add "tags" as something that
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	132 # is always a tuple, even when it's not a combination.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	133 if (self.taxonomy.is_multiple and
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	134 self.taxonomy.name != self.taxonomy.term_name):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	135 mult_val = tax_terms
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	136 if not is_combination:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	137 mult_val = (mult_val,)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	138 ctx.custom_data[self.taxonomy.name] = mult_val
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	139
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	140 def _getTaxonomyTerms(self, route_params):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	141 # Get the individual slugified terms from the route metadata.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	142 all_values = route_params.get(self.taxonomy.term_name)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	143 if all_values is None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	144 raise Exception("'%s' values couldn't be found in route metadata" %
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	145 self.taxonomy.term_name)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	146
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	147 # If it's a "multiple" taxonomy, we need to potentially split the
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	148 # route value into the individual terms (_e.g._ when listing all pages
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	149 # that have 2 given tags, we need to get each of those 2 tags).
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150 if self.taxonomy.is_multiple:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	151 sep = self.taxonomy.separator
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	152 if sep in all_values:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	153 return tuple(all_values.split(sep)), True
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	154 # Not a "multiple" taxonomy, so there's only the one value.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	155 return all_values, False
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	156
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	157 def _setTaxonomyFilter(self, ctx, term_value, is_combination):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	158 # Set up the filter that will check the pages' terms.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	159 flt = PaginationFilter()
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	160 flt.addClause(HasTaxonomyTermsFilterClause(
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	161 self.taxonomy, self.slugifier.mode, term_value, is_combination))
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	162 ctx.pagination_filter = flt
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	163
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	164 def onRouteFunctionUsed(self, route_params):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	165 # Get the values, and slugify them appropriately.
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	166 # If this is a "multiple" taxonomy, `values` will be a tuple of
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	167 # terms. If not, `values` will just be a term.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	168 values = route_params[self.taxonomy.term_name]
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	169 tax_is_multiple = self.taxonomy.is_multiple
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	170 if tax_is_multiple:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	171 slugified_values = self.slugifyMultiple((str(v) for v in values))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	172 route_val = self.taxonomy.separator.join(slugified_values)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	173 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	174 slugified_values = self.slugify(str(values))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	175 route_val = slugified_values
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	176
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	177 # We need to register this use of a taxonomy term.
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	178 # Because the render info gets serialized across bake worker
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	179 # processes, we can only use basic JSON-able structures, which
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	180 # excludes `set`... hence the awkward use of `list`.
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	181 # Also, note that the tuples we're putting in there will be
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	182 # transformed into lists so we'll have to convert back.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	183 rcs = self.app.env.render_ctx_stack
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	184 ri = rcs.current_ctx.render_info
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	185 utt = ri.get('used_taxonomy_terms')
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	186 if utt is None:
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	187 ri['used_taxonomy_terms'] = [slugified_values]
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	188 else:
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	189 if slugified_values not in utt:
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	190 utt.append(slugified_values)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	191
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	192 # Put the slugified values in the route metadata so they're used to
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	193 # generate the URL.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	194 route_params[self.taxonomy.term_name] = route_val
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	195
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	196
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	197 class HasTaxonomyTermsFilterClause(SettingFilterClause):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	198 def __init__(self, taxonomy, slugify_mode, value, is_combination):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	199 super().__init__(taxonomy.setting_name, value)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	200 self._taxonomy = taxonomy
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	201 self._is_combination = is_combination
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	202 self._slugifier = _Slugifier(taxonomy, slugify_mode)
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	203 if taxonomy.is_multiple:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	204 self.pageMatches = self._pageMatchesAny
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	205 else:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	206 self.pageMatches = self._pageMatchesSingle
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	207
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	208 def _pageMatchesAny(self, fil, page):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	209 # Multiple taxonomy, i.e. it supports multiple terms, like tags.
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	210 page_values = page.config.get(self.name)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	211 if page_values is None or not isinstance(page_values, list):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	212 return False
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	213
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	214 page_set = set(map(self._slugifier.slugify, page_values))
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	215 if self._is_combination:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	216 # Multiple taxonomy, and multiple terms to match. Check that
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	217 # the ones to match are all in the page's terms.
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	218 value_set = set(self.value)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	219 return value_set.issubset(page_set)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	220 else:
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	221 # Multiple taxonomy, one term to match.
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	222 return self.value in page_set
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	223
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	224 def _pageMatchesSingle(self, fil, page):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	225 # Single taxonomy. Just compare the values.
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	226 page_value = page.config.get(self.name)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	227 if page_value is None:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	228 return False
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	229 page_value = self._slugifier.slugify(page_value)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	230 return page_value == self.value
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	231
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	232
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	233 def _get_taxonomy(app, tax_name):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	234 tax_config = app.config.get('site/taxonomies/' + tax_name)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	235 if tax_config is None:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	236 raise ConfigurationError("No such taxonomy: %s" % tax_name)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	237 return Taxonomy(tax_name, tax_config)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	238
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	239
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	240 def _get_slugifier(app, taxonomy, slugify_mode=None):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	241 if slugify_mode is None:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	242 slugify_mode = app.config.get('site/slugify_mode', 'encode')
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	243 sm = _parse_slugify_mode(slugify_mode)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	244 return _Slugifier(taxonomy, sm)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	245
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	246
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	247 class TaxonomyPipelineRecordEntry(PagePipelineRecordEntry):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	248 def __init__(self):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	249 super().__init__()
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	250 self.term = None
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	251
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	252
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	253 class TaxonomyPipeline(ContentPipeline):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	254 PIPELINE_NAME = 'taxonomy'
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	255 PASS_NUM = 10
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	256 RECORD_ENTRY_CLASS = TaxonomyPipelineRecordEntry
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	257
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	258 def __init__(self, source, ctx):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	259 if not isinstance(source, TaxonomySource):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	260 raise Exception("The taxonomy pipeline only supports taxonomy "
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	261 "content sources.")
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	262
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	263 super().__init__(source, ctx)
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	264 self.inner_source = source.inner_source
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	265 self.taxonomy = source.taxonomy
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	266 self.slugifier = source.slugifier
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	267 self._tpl_name = source.config['template']
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	268 self._analyzer = None
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	269 self._pagebaker = None
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	270
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	271 def initialize(self):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	272 self._pagebaker = PageBaker(self.app,
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	273 self.ctx.out_dir,
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	274 force=self.ctx.force)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	275 self._pagebaker.startWriterQueue()
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	276
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	277 def shutdown(self):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	278 self._pagebaker.stopWriterQueue()
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	279
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	280 def createJobs(self, ctx):
876 d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	281 logger.debug("Caching template page for taxonomy '%s'." %
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	282 self.taxonomy.name)
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	283 page = self.app.getPage(self.source, ContentItem('_index', {}))
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	284 page._load()
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	285
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	286 logger.debug("Building '%s' taxonomy pages for source: %s" %
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	287 (self.taxonomy.name, self.inner_source.name))
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	288 self._analyzer = _TaxonomyTermsAnalyzer(self, ctx.record_histories)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	289 self._analyzer.analyze()
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	290
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	291 logger.debug("Queuing %d '%s' jobs." %
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	292 (len(self._analyzer.dirty_slugified_terms),
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	293 self.taxonomy.name))
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	294 jobs = []
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	295 rec_fac = self.createRecordEntry
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	296 current_record = ctx.current_record
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	297
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	298 for slugified_term in self._analyzer.dirty_slugified_terms:
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	299 item_spec = '_index'
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	300 record_entry_spec = '_index[%s]' % slugified_term
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	301
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	302 jobs.append(create_job(self, item_spec,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	303 term=slugified_term,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	304 record_entry_spec=record_entry_spec))
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	305
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	306 entry = rec_fac(record_entry_spec)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	307 current_record.addEntry(entry)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	308
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	309 if len(jobs) > 0:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	310 return jobs
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	311 return None
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	312
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	313 def run(self, job, ctx, result):
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	314 term = job['term']
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	315 content_item = ContentItem('_index',
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	316 {'term': term,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	317 'route_params': {
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	318 self.taxonomy.term_name: term}
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	319 })
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	320 page = Page(self.source, content_item)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	321
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	322 logger.debug("Rendering '%s' page: %s" %
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	323 (self.taxonomy.name, page.source_metadata['term']))
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	324 prev_entry = ctx.previous_entry
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	325 rdr_subs = self._pagebaker.bake(page, prev_entry)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	326
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	327 add_page_job_result(result)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	328 result['subs'] = rdr_subs
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	329 result['term'] = page.source_metadata['term']
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	330
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	331 def handleJobResult(self, result, ctx):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	332 existing = ctx.record_entry
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	333 merge_job_result_into_record_entry(existing, result)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	334 existing.term = result['term']
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	335
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	336 def postJobRun(self, ctx):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	337 # We create bake entries for all the terms that were not dirty.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	338 # This is because otherwise, on the next incremental bake, we wouldn't
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	339 # find any entry for those things, and figure that we need to delete
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	340 # their outputs.
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	341 analyzer = self._analyzer
882 acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	342 record = ctx.record_history.current
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	343 for prev, cur in ctx.record_history.diffs:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	344 # Only consider entries that don't have any current version
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	345 # (i.e. they weren't baked just now).
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	346 if prev and not cur:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	347 t = prev.term
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	348 if analyzer.isKnownSlugifiedTerm(t):
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	349 logger.debug("Creating unbaked entry for '%s' term: %s" %
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	350 (self.taxonomy.name, t))
882 acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	351 cur = copy.deepcopy(prev)
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	352 cur.flags = \
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	353 PagePipelineRecordEntry.FLAG_COLLAPSED_FROM_LAST_RUN
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	354 record.addEntry(cur)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	355 else:
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	356 logger.debug("Term '%s' in '%s' isn't used anymore." %
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	357 (t, self.taxonomy.name))
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	358
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	359
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	360 class _TaxonomyTermsAnalyzer(object):
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	361 def __init__(self, pipeline, record_histories):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	362 self.pipeline = pipeline
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	363 self.record_histories = record_histories
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	364 self._all_terms = {}
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	365 self._single_dirty_slugified_terms = set()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	366 self._all_dirty_slugified_terms = None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	367
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	368 @property
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	369 def dirty_slugified_terms(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	370 """ Returns the slugified terms that have been 'dirtied' during
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	371 this bake.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	372 """
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	373 return self._all_dirty_slugified_terms
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	374
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	375 def isKnownSlugifiedTerm(self, term):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	376 """ Returns whether the given slugified term has been seen during
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	377 this bake.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	378 """
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	379 return term in self._all_terms
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	380
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	381 def analyze(self):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	382 # Build the list of terms for our taxonomy, and figure out which ones
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	383 # are 'dirty' for the current bake.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	384 #
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	385 # Remember all terms used.
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	386 source = self.pipeline.inner_source
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	387 taxonomy = self.pipeline.taxonomy
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	388 slugifier = self.pipeline.slugifier
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	389
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	390 record_name = get_record_name_for_source(source)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	391 current_records = self.record_histories.current
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	392 cur_rec = current_records.getRecord(record_name)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	393 for cur_entry in cur_rec.getEntries():
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	394 if not cur_entry.was_overriden:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	395 cur_terms = cur_entry.config.get(taxonomy.setting_name)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	396 if cur_terms:
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	397 if not taxonomy.is_multiple:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	398 self._addTerm(
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	399 slugifier, cur_entry.item_spec, cur_terms)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	400 else:
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	401 self._addTerms(
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	402 slugifier, cur_entry.item_spec, cur_terms)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	403
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	404 # Re-bake all taxonomy terms that include new or changed pages, by
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	405 # marking them as 'dirty'.
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	406 history = self.record_histories.getHistory(record_name).copy()
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	407 history.build()
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	408 for prev_entry, cur_entry in history.diffs:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	409 entries = [cur_entry]
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	410 if prev_entry:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	411 entries.append(prev_entry)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	412
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	413 for e in entries:
896 85d2b386b971 fix: Don't crash during some iterative builds. Ludovic Chabant <ludovic@chabant.com> parents: 882 diff changeset	414 if e and e.was_any_sub_baked:
882 acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	415 entry_terms = e.config.get(taxonomy.setting_name)
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	416 if entry_terms:
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	417 if not taxonomy.is_multiple:
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	418 self._single_dirty_slugified_terms.add(
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	419 slugifier.slugify(entry_terms))
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	420 else:
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	421 self._single_dirty_slugified_terms.update(
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	422 (slugifier.slugify(t)
acd9c3e8533f bake: Correctly setup unbaked entries for taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	423 for t in entry_terms))
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	424
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	425 self._all_dirty_slugified_terms = list(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	426 self._single_dirty_slugified_terms)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	427 logger.debug("Gathered %d dirty taxonomy terms",
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	428 len(self._all_dirty_slugified_terms))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	429
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	430 # Re-bake the combination pages for terms that are 'dirty'.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	431 # We make all terms into tuple, even those that are not actual
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	432 # combinations, so that we have less things to test further down the
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	433 # line.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	434 #
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	435 # Add the combinations to that list. We get those combinations from
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	436 # wherever combinations were used, so they're coming from the
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	437 # `onRouteFunctionUsed` method. And because combinations can be used
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	438 # by any page in the website (anywhere someone can ask for an URL
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	439 # to the combination page), it means we check all the records, not
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	440 # just the record for our source.
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	441 if taxonomy.is_multiple:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	442 known_combinations = set()
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	443 for rec in current_records.records:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	444 # Cheap way to test if a record contains entries that
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	445 # are sub-types of a page entry: test the first one.
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	446 first_entry = next(iter(rec.getEntries()), None)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	447 if (first_entry is None or
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	448 not isinstance(first_entry, PagePipelineRecordEntry)):
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	449 continue
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	450
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	451 for cur_entry in rec.getEntries():
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	452 used_terms = _get_all_entry_taxonomy_terms(cur_entry)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	453 for terms in used_terms:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	454 if len(terms) > 1:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 896 diff changeset	455 known_combinations.add(terms)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	456
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	457 dcc = 0
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	458 for terms in known_combinations:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	459 if not self._single_dirty_slugified_terms.isdisjoint(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	460 set(terms)):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	461 self._all_dirty_slugified_terms.append(
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	462 taxonomy.separator.join(terms))
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	463 dcc += 1
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	464 logger.debug("Gathered %d term combinations, with %d dirty." %
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	465 (len(known_combinations), dcc))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	466
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	467 def _addTerms(self, slugifier, item_spec, terms):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	468 for t in terms:
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	469 self._addTerm(slugifier, item_spec, t)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	470
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	471 def _addTerm(self, slugifier, item_spec, term):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	472 st = slugifier.slugify(term)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	473 orig_terms = self._all_terms.setdefault(st, [])
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	474 if orig_terms and orig_terms[0] != term:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	475 logger.warning(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	476 "Term '%s' in '%s' is slugified to '%s' which conflicts with "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	477 "previously existing '%s'. The two will be merged." %
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	478 (term, item_spec, st, orig_terms[0]))
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	479 orig_terms.append(term)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	480
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	481
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	482 def _get_all_entry_taxonomy_terms(entry):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	483 res = set()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	484 for o in entry.subs:
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	485 pinfo = o['render_info']
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	486 terms = pinfo.get('used_taxonomy_terms')
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	487 if terms:
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	488 res \|= set([tuple(t) for t in terms])
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	489 return res
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	490
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	491
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	492 class _Slugifier(object):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	493 def __init__(self, taxonomy, mode):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	494 self.taxonomy = taxonomy
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	495 self.mode = mode
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	496
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	497 def slugifyMultiple(self, terms):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	498 return tuple(map(self.slugify, terms))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	499
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	500 def slugify(self, term):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	501 if self.mode & SLUGIFY_TRANSLITERATE:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	502 term = unidecode.unidecode(term)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	503 if self.mode & SLUGIFY_LOWERCASE:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	504 term = term.lower()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	505 if self.mode & SLUGIFY_DOT_TO_DASH:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	506 term = re_first_dot_to_dash.sub('', term)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	507 term = re_dot_to_dash.sub('-', term)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	508 if self.mode & SLUGIFY_SPACE_TO_DASH:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	509 term = re_space_to_dash.sub('-', term)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	510 return term
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	511
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	512
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	513 def _parse_slugify_mode(value):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	514 mapping = {
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	515 'encode': SLUGIFY_ENCODE,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	516 'transliterate': SLUGIFY_TRANSLITERATE,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	517 'lowercase': SLUGIFY_LOWERCASE,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	518 'dot_to_dash': SLUGIFY_DOT_TO_DASH,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	519 'space_to_dash': SLUGIFY_SPACE_TO_DASH}
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	520 mode = 0
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	521 for v in value.split(','):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	522 f = mapping.get(v.strip())
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	523 if f is None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	524 if v == 'iconv':
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	525 raise Exception("'iconv' is not supported as a slugify mode "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	526 "in PieCrust2. Use 'transliterate'.")
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	527 raise Exception("Unknown slugify flag: %s" % v)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	528 mode \|= f
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	529 return mode
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	530

Mercurial > piecrust2

annotate piecrust/sources/taxonomy.py @ 1114:8af2ea1f5c34