piecrust2: piecrust/baking/baker.py annotate

annotate piecrust/baking/baker.py @ 72:ecee3e8f35e5

Fix `setuptools` install.

author	Ludovic Chabant <ludovic@chabant.com>
date	Sat, 30 Aug 2014 09:46:37 -0700
parents	2fec3ee1298f
children	3471ffa059b2

rev	line source
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import time
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os.path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import codecs
25 65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	4 import shutil
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import hashlib
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 import logging
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	7 import threading
25 65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	8 import urllib.request, urllib.error, urllib.parse
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	9 from queue import Queue, Empty
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	10 from piecrust.baking.records import TransitionalBakeRecord, BakeRecordPageEntry
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	11 from piecrust.chefutil import format_timed, log_friendly_exception
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12 from piecrust.data.filters import (PaginationFilter, HasFilterClause,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13 IsFilterClause, AndBooleanClause)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	14 from piecrust.processing.base import ProcessorPipeline
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	15 from piecrust.rendering import PageRenderingContext, render_page
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16 from piecrust.sources.base import (PageFactory,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17 REALM_NAMES, REALM_USER, REALM_THEME)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	23 class PageBaker(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24 def __init__(self, app, out_dir, force=False, record=None,
25 65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	25 copy_assets=True):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26 self.app = app
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27 self.out_dir = out_dir
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 self.force = force
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29 self.record = record
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	30 self.force = force
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31 self.copy_assets = copy_assets
25 65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	32 self.site_root = app.config.get('site/root')
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 self.pretty_urls = app.config.get('site/pretty_urls')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34 self.pagination_suffix = app.config.get('site/pagination_suffix')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 def getOutputUri(self, uri, num):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37 suffix = self.pagination_suffix.replace('%num%', str(num))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	38 if self.pretty_urls:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	39 # Output will be:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	40 # - `uri/name`
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41 # - `uri/name/2`
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42 # - `uri/name.ext`
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	43 # - `uri/name.ext/2`
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44 if num <= 1:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	45 return uri
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	46 return uri + suffix
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	47 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	48 # Output will be:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 # - `uri/name.html`
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50 # - `uri/name/2.html`
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51 # - `uri/name.ext`
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52 # - `uri/name/2.ext`
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53 if uri == '/':
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54 if num <= 1:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	55 return '/'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	56 return '/' + suffix.lstrip('/')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	57 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58 if num <= 1:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 return uri
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	60 #TODO: watch out for tags with dots in them.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	61 base_uri, ext = os.path.splitext(uri)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62 return base_uri + suffix + ext
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64 def getOutputPath(self, uri):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	65 bake_path = [self.out_dir]
6 f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: 5 diff changeset	66 decoded_uri = urllib.parse.unquote(uri.lstrip('/'))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	67 if self.pretty_urls:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 bake_path.append(decoded_uri)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69 bake_path.append('index.html')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	70 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71 name, ext = os.path.splitext(decoded_uri)
6 f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: 5 diff changeset	72 if decoded_uri == '':
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: 5 diff changeset	73 bake_path.append('index.html')
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: 5 diff changeset	74 elif ext:
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75 bake_path.append(decoded_uri)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	77 bake_path.append(decoded_uri + '.html')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	78
11 617191dec18e Fixes for Windows, make `findPagePath` return a ref path. Ludovic Chabant <ludovic@chabant.com> parents: 7 diff changeset	79 return os.path.normpath(os.path.join(*bake_path))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	81 def bake(self, factory, route, taxonomy_name=None, taxonomy_term=None):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	82 pagination_filter = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	83 custom_data = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84 if taxonomy_name and taxonomy_term:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	85 # Must bake a taxonomy listing page... we'll have to add a
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	86 # pagination filter for only get matching posts, and the output
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	87 # URL will be a bit different.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	88 tax = self.app.getTaxonomy(taxonomy_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	89 pagination_filter = PaginationFilter()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	90 if tax.is_multiple:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	91 if isinstance(taxonomy_term, tuple):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	92 abc = AndBooleanClause()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	93 for t in taxonomy_term:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	94 abc.addClause(HasFilterClause(taxonomy_name, t))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	95 pagination_filter.addClause(abc)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	96 slugified_term = '/'.join(taxonomy_term)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	97 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	98 pagination_filter.addClause(HasFilterClause(taxonomy_name,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	99 taxonomy_term))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	100 slugified_term = taxonomy_term
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	101 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	102 pagination_filter.addClause(IsFilterClause(taxonomy_name,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	103 taxonomy_term))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	104 slugified_term = taxonomy_term
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	105 custom_data = {tax.term_name: taxonomy_term}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	106 uri = route.getUri({tax.term_name: slugified_term})
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	107 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	108 # Normal page bake.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	109 uri = route.getUri(factory.metadata)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	110
50 2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	111 override = self.record.getOverrideEntry(factory, uri)
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	112 if override is not None:
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	113 override_source = self.app.getSource(override.source_name)
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	114 if override_source.realm == factory.source.realm:
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	115 raise Exception(
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	116 "Page '%s' maps to URL '%s' but is overriden by page"
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	117 "'%s:%s'." % (factory.ref_spec, uri,
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	118 override.source_name, override.rel_path))
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	119 logger.debug("'%s' [%s] is overriden by '%s:%s'. Skipping" %
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	120 (factory.ref_spec, uri, override.source_name,
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	121 override.rel_path))
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	122 entry = BakeRecordPageEntry()
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	123 entry.path = factory.path
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	124 entry.rel_path = factory.rel_path
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	125 entry.source_name = factory.source.name
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	126 entry.was_overriden = True
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	127
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	128 if self.record:
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	129 self.record.addEntry(entry)
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	130
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	131 return entry
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	132
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	133 cur_sub = 1
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	134 has_more_subs = True
50 2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	135 page = factory.buildPage()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	136 cur_record_entry = BakeRecordPageEntry(page)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	137 cur_record_entry.taxonomy_name = taxonomy_name
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	138 cur_record_entry.taxonomy_term = taxonomy_term
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	139 prev_record_entry = self.record.getPreviousEntry(page, taxonomy_name,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	140 taxonomy_term)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	141
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	142 logger.debug("Baking '%s'..." % uri)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	143 while has_more_subs:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	144 sub_uri = self.getOutputUri(uri, cur_sub)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	145 out_path = self.getOutputPath(sub_uri)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	146
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	147 # Check for up-to-date outputs.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	148 do_bake = True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	149 if not self.force and prev_record_entry:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150 try:
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	151 in_path_time = page.path_mtime
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	152 out_path_time = os.path.getmtime(out_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	153 if out_path_time > in_path_time:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	154 do_bake = False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	155 except OSError:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	156 # File doesn't exist, we'll need to bake.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	157 pass
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	158
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	159 # If this page didn't bake because it's already up-to-date.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	160 # Keep trying for as many subs as we know this page has.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	161 if not do_bake:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	162 if (prev_record_entry is not None and
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	163 prev_record_entry.num_subs < cur_sub):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	164 logger.debug("")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	165 cur_sub += 1
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	166 has_more_subs = True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	167 logger.debug(" %s is up to date, skipping to next "
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	168 "sub-page." % out_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	169 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	170
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	171 # We don't know how many subs to expect... just skip.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	172 logger.debug(" %s is up to date, skipping bake." % out_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	173 break
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	174
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	175 # All good, proceed.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	176 try:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	177 logger.debug(" p%d -> %s" % (cur_sub, out_path))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	178 ctx, rp = self._bakeSingle(page, sub_uri, cur_sub, out_path,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	179 pagination_filter, custom_data)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	180 except Exception as ex:
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	181 if self.app.debug:
fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	182 logger.exception(ex)
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	183 raise Exception("Error baking page '%s' for URL '%s'." %
11 617191dec18e Fixes for Windows, make `findPagePath` return a ref path. Ludovic Chabant <ludovic@chabant.com> parents: 7 diff changeset	184 (page.ref_spec, uri)) from ex
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	185
25 65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	186 # Copy page assets.
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	187 if (cur_sub == 1 and self.copy_assets and
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	188 ctx.used_assets is not None):
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	189 if self.pretty_urls:
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	190 out_assets_dir = os.path.dirname(out_path)
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	191 else:
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	192 out_assets_dir, out_name = os.path.split(out_path)
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	193 if sub_uri != self.site_root:
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	194 out_name_noext, _ = os.path.splitext(out_name)
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	195 out_assets_dir += out_name_noext
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	196
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	197 logger.debug("Copying page assets to: %s" % out_assets_dir)
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	198 if not os.path.isdir(out_assets_dir):
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	199 os.makedirs(out_assets_dir, 0o755)
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	200 for ap in ctx.used_assets._getAssetPaths():
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	201 dest_ap = os.path.join(out_assets_dir, os.path.basename(ap))
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	202 logger.debug(" %s -> %s" % (ap, dest_ap))
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	203 shutil.copy(ap, dest_ap)
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	204
65ae19c4e8a3 Copy page assets to bake output, use correct slashes when serving assets. Ludovic Chabant <ludovic@chabant.com> parents: 23 diff changeset	205 # Record what we did and figure out if we have more work.
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	206 cur_record_entry.out_uris.append(sub_uri)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	207 cur_record_entry.out_paths.append(out_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	208 cur_record_entry.used_source_names \|= ctx.used_source_names
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	209 cur_record_entry.used_taxonomy_terms \|= ctx.used_taxonomy_terms
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	210
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	211 has_more_subs = False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	212 if ctx.used_pagination is not None:
23 923699e816d0 Don't try to get the name of a source that doesn't have one. Ludovic Chabant <ludovic@chabant.com> parents: 11 diff changeset	213 cur_record_entry.addUsedSource(ctx.used_pagination._source)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	214 if ctx.used_pagination.has_more:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	215 cur_sub += 1
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	216 has_more_subs = True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	217
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	218 if self.record:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	219 self.record.addEntry(cur_record_entry)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	220
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	221 return cur_record_entry
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	222
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	223 def _bakeSingle(self, page, sub_uri, num, out_path,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	224 pagination_filter=None, custom_data=None):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	225 ctx = PageRenderingContext(page, sub_uri)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	226 ctx.page_num = num
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	227 if pagination_filter:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	228 ctx.pagination_filter = pagination_filter
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	229 if custom_data:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	230 ctx.custom_data = custom_data
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	231
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	232 rp = render_page(ctx)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	233
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	234 out_dir = os.path.dirname(out_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	235 if not os.path.isdir(out_dir):
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	236 os.makedirs(out_dir, 0o755)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	237
7 343d08ef5668 More PieCrust 3 fixes, and a couple of miscellaneous bug fixes. Ludovic Chabant <ludovic@chabant.com> parents: 6 diff changeset	238 with codecs.open(out_path, 'w', 'utf8') as fp:
6 f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: 5 diff changeset	239 fp.write(rp.content)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	240
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	241 return ctx, rp
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	242
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	243
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	244 class Baker(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	245 def __init__(self, app, out_dir=None, force=False, portable=False,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	246 no_assets=False):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	247 self.app = app
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	248 self.out_dir = out_dir or os.path.join(app.root_dir, '_counter')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	249 self.force = force
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	250 self.portable = portable
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	251 self.no_assets = no_assets
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	252 self.num_workers = app.config.get('baker/workers') or 4
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	253
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	254 # Remember what taxonomy pages we should skip
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	255 # (we'll bake them repeatedly later with each taxonomy term)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	256 self.taxonomy_pages = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	257 logger.debug("Gathering taxonomy page paths:")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	258 for tax in self.app.taxonomies:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	259 for src in self.app.sources:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	260 path = tax.resolvePagePath(src.name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	261 if path is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	262 self.taxonomy_pages.append(path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	263 logger.debug(" - %s" % path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	264
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	265 def bake(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	266 logger.debug(" Bake Output: %s" % self.out_dir)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	267 logger.debug(" Root URL: %s" % self.app.config.get('site/root'))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	268
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	269 # Get into bake mode.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	270 start_time = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	271 self.app.config.set('baker/is_baking', True)
32 43091c9837bf Fix problems with asset URLs. Ludovic Chabant <ludovic@chabant.com> parents: 25 diff changeset	272 self.app.env.base_asset_url_format = '%uri%'
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	273
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	274 # Make sure the output directory exists.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	275 if not os.path.isdir(self.out_dir):
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	276 os.makedirs(self.out_dir, 0o755)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	277
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	278 # Load/create the bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	279 record = TransitionalBakeRecord()
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 42 diff changeset	280 record_cache = self.app.cache.getCache('baker')
7 343d08ef5668 More PieCrust 3 fixes, and a couple of miscellaneous bug fixes. Ludovic Chabant <ludovic@chabant.com> parents: 6 diff changeset	281 record_name = (hashlib.md5(self.out_dir.encode('utf8')).hexdigest() +
343d08ef5668 More PieCrust 3 fixes, and a couple of miscellaneous bug fixes. Ludovic Chabant <ludovic@chabant.com> parents: 6 diff changeset	282 '.record')
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	283 if not self.force and record_cache.has(record_name):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	284 t = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	285 record.loadPrevious(record_cache.getCachePath(record_name))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	286 logger.debug(format_timed(t, 'loaded previous bake record',
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	287 colored=False));
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	288
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	289 # Figure out if we need to clean the cache because important things
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	290 # have changed.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	291 self._handleCacheValidity(record)
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	292
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	293 # Gather all sources by realm -- we're going to bake each realm
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	294 # separately so we can handle "overlaying" (i.e. one realm overrides
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	295 # another realm's pages).
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	296 sources_by_realm = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	297 for source in self.app.sources:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	298 srclist = sources_by_realm.setdefault(source.realm, [])
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	299 srclist.append(source)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	300
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	301 # Bake the realms.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	302 realm_list = [REALM_USER, REALM_THEME]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	303 for realm in realm_list:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	304 srclist = sources_by_realm.get(realm)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	305 if srclist is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	306 self._bakeRealm(record, realm, srclist)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	307
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	308 # Bake taxonomies.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	309 self._bakeTaxonomies(record)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	310
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	311 # Bake the assets.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	312 if not self.no_assets:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	313 self._bakeAssets(record)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	314
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	315 # Save the bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	316 t = time.clock()
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	317 record.current.bake_time = time.time()
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	318 record.collapseRecords()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	319 record.saveCurrent(record_cache.getCachePath(record_name))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	320 logger.debug(format_timed(t, 'saved bake record', colored=False))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	321
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	322 # All done.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	323 self.app.config.set('baker/is_baking', False)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	324 logger.info('-------------------------');
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	325 logger.info(format_timed(start_time, 'done baking'));
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	326
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	327 def _handleCacheValidity(self, record):
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	328 start_time = time.clock()
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	329
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	330 reason = None
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	331 if self.force:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	332 reason = "ordered to"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	333 elif not self.app.config.get('__cache_valid'):
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	334 # The configuration file was changed, or we're running a new
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	335 # version of the app.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	336 reason = "not valid anymore"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	337 elif not record.previous.bake_time:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	338 # We have no valid previous bake record.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	339 reason = "need bake record regeneration"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	340 else:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	341 # Check if any template has changed since the last bake. Since
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	342 # there could be some advanced conditional logic going on, we'd
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	343 # better just force a bake from scratch if that's the case.
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	344 max_time = 0
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	345 for d in self.app.templates_dirs:
42 9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	346 for dpath, _, filenames in os.walk(d):
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	347 for fn in filenames:
42 9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	348 full_fn = os.path.join(dpath, fn)
9e058e221108 Fix a crash when checking for timestamps on template files. Ludovic Chabant <ludovic@chabant.com> parents: 39 diff changeset	349 max_time = max(max_time, os.path.getmtime(full_fn))
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	350 if max_time >= record.previous.bake_time:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	351 reason = "templates modified"
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	352
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	353 if reason is not None:
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 42 diff changeset	354 cache_dir = self.app.cache.getCacheDir('baker')
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	355 if os.path.isdir(cache_dir):
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 42 diff changeset	356 logger.debug("Cleaning baker cache: %s" % cache_dir)
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	357 shutil.rmtree(cache_dir)
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 42 diff changeset	358 self.force = True
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	359 logger.info(format_timed(start_time,
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	360 "cleaned cache (reason: %s)" % reason))
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	361 else:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	362 logger.debug(format_timed(start_time, "cache is assumed valid",
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	363 colored=False))
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	364
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	365 def _bakeRealm(self, record, realm, srclist):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	366 # Gather all page factories from the sources and queue them
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	367 # for the workers to pick up. Just skip taxonomy pages for now.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	368 logger.debug("Baking realm %s" % REALM_NAMES[realm])
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	369 pool, queue, abort = self._createWorkerPool(record, self.num_workers)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	370
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	371 for source in srclist:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	372 factories = source.getPageFactories()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	373 for fac in factories:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	374 if fac.path in self.taxonomy_pages:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	375 logger.debug("Skipping taxonomy page: %s:%s" %
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	376 (source.name, fac.ref_spec))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	377 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	378
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	379 route = self.app.getRoute(source.name, fac.metadata)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	380 if route is None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	381 logger.error("Can't get route for page: %s" % fac.ref_spec)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	382 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	383
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	384 logger.debug("Queuing: %s" % fac.ref_spec)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	385 queue.put_nowait(BakeWorkerJob(fac, route))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	386
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	387 self._waitOnWorkerPool(pool, abort)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	388
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	389 def _bakeTaxonomies(self, record):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	390 logger.debug("Baking taxonomies")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	391
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	392 # Let's see all the taxonomy terms for which we must bake a
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	393 # listing page... first, pre-populate our big map of used terms.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	394 buckets = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	395 tax_names = [t.name for t in self.app.taxonomies]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	396 source_names = [s.name for s in self.app.sources]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	397 for sn in source_names:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	398 source_taxonomies = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	399 buckets[sn] = source_taxonomies
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	400 for tn in tax_names:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	401 source_taxonomies[tn] = set()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	402
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	403 # Now see which ones are 'dirty' based on our bake record.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	404 logger.debug("Gathering dirty taxonomy terms")
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	405 for prev_entry, cur_entry in record.transitions.values():
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	406 for tax in self.app.taxonomies:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	407 changed_terms = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	408 # Re-bake all taxonomy pages that include new or changed
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	409 # pages.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	410 if not prev_entry and cur_entry and cur_entry.was_baked:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	411 changed_terms = cur_entry.config.get(tax.name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	412 elif prev_entry and cur_entry and cur_entry.was_baked:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	413 changed_terms = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	414 prev_terms = prev_entry.config.get(tax.name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	415 cur_terms = cur_entry.config.get(tax.name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	416 if tax.is_multiple:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	417 if prev_terms is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	418 changed_terms += prev_terms
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	419 if cur_terms is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	420 changed_terms += cur_terms
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	421 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	422 if prev_terms is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	423 changed_terms.append(prev_terms)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	424 if cur_terms is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	425 changed_terms.append(cur_terms)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	426 if changed_terms is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	427 if not isinstance(changed_terms, list):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	428 changed_terms = [changed_terms]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	429 buckets[cur_entry.source_name][tax.name] \|= (
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	430 set(changed_terms))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	431
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	432 # Re-bake the combination pages for terms that are 'dirty'.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	433 known_combinations = set()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	434 logger.debug("Gathering dirty term combinations")
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	435 for prev_entry, cur_entry in record.transitions.values():
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	436 if cur_entry:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	437 known_combinations \|= cur_entry.used_taxonomy_terms
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	438 elif prev_entry:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	439 known_combinations \|= prev_entry.used_taxonomy_terms
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	440 for sn, tn, terms in known_combinations:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	441 changed_terms = buckets[sn][tn]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	442 if not changed_terms.isdisjoint(set(terms)):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	443 changed_terms.add(terms)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	444
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	445 # Start baking those terms.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	446 pool, queue, abort = self._createWorkerPool(record, self.num_workers)
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	447 for source_name, source_taxonomies in buckets.items():
474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	448 for tax_name, terms in source_taxonomies.items():
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	449 if len(terms) == 0:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	450 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	451
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	452 logger.debug("Baking '%s' for source '%s': %s" %
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	453 (tax_name, source_name, terms))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	454 tax = self.app.getTaxonomy(tax_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	455 route = self.app.getTaxonomyRoute(tax_name, source_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	456 tax_page_ref = tax.getPageRef(source_name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	457 if not tax_page_ref.exists:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	458 logger.debug("No taxonomy page found at '%s', skipping." %
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	459 tax.page_ref)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	460 continue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	461
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	462 tax_page_source = tax_page_ref.source
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	463 tax_page_rel_path = tax_page_ref.rel_path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	464 logger.debug("Using taxonomy page: %s:%s" %
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	465 (tax_page_source.name, tax_page_rel_path))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	466
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	467 for term in terms:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	468 fac = PageFactory(tax_page_source, tax_page_rel_path,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	469 {tax.term_name: term})
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	470 logger.debug("Queuing: %s [%s, %s]" %
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	471 (fac.ref_spec, tax_name, term))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	472 queue.put_nowait(
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	473 BakeWorkerJob(fac, route, tax_name, term))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	474
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	475 self._waitOnWorkerPool(pool, abort)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	476
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	477 def _bakeAssets(self, record):
36 485682a6de50 New site layout support. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	478 mounts = self.app.assets_dirs
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	479 baker_params = self.app.config.get('baker') or {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	480 skip_patterns = baker_params.get('skip_patterns')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	481 force_patterns = baker_params.get('force_patterns')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	482 proc = ProcessorPipeline(
36 485682a6de50 New site layout support. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	483 self.app, mounts, self.out_dir, force=self.force,
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	484 skip_patterns=skip_patterns, force_patterns=force_patterns,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	485 num_workers=self.num_workers)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	486 proc.run()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	487
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	488 def _createWorkerPool(self, record, pool_size=4):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	489 pool = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	490 queue = Queue()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	491 abort = threading.Event()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	492 for i in range(pool_size):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	493 ctx = BakeWorkerContext(self.app, self.out_dir, self.force,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	494 record, queue, abort)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	495 worker = BakeWorker(i, ctx)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	496 worker.start()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	497 pool.append(worker)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	498 return pool, queue, abort
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	499
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	500 def _waitOnWorkerPool(self, pool, abort):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	501 for w in pool:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	502 w.join()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	503 if abort.is_set():
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	504 excs = [w.abort_exception for w in pool
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	505 if w.abort_exception is not None]
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	506 logger.error("%s errors" % len(excs))
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	507 if self.app.debug:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	508 for e in excs:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	509 logger.exception(e)
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	510 else:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	511 for e in excs:
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	512 log_friendly_exception(logger, e)
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	513 raise Exception("Baking was aborted due to errors.")
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	514
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	515
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	516 class BakeWorkerContext(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	517 def __init__(self, app, out_dir, force, record, work_queue,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	518 abort_event):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	519 self.app = app
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	520 self.out_dir = out_dir
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	521 self.force = force
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	522 self.record = record
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	523 self.work_queue = work_queue
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	524 self.abort_event = abort_event
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	525
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	526
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	527 class BakeWorkerJob(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	528 def __init__(self, factory, route, taxonomy_name=None, taxonomy_term=None):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	529 self.factory = factory
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	530 self.route = route
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	531 self.taxonomy_name = taxonomy_name
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	532 self.taxonomy_term = taxonomy_term
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	533
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	534 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	535 def source(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	536 return self.factory.source
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	537
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	538
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	539 class BakeWorker(threading.Thread):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	540 def __init__(self, wid, ctx):
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	541 super(BakeWorker, self).__init__(name=('worker%d' % wid))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	542 self.wid = wid
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	543 self.ctx = ctx
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	544 self.abort_exception = None
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	545 self._page_baker = PageBaker(ctx.app, ctx.out_dir, ctx.force,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	546 ctx.record)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	547
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	548 def run(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	549 while(not self.ctx.abort_event.is_set()):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	550 try:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	551 job = self.ctx.work_queue.get(True, 0.1)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	552 except Empty:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	553 logger.debug("[%d] No more work... shutting down." % self.wid)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	554 break
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	555
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	556 try:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	557 self._unsafeRun(job)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	558 logger.debug("[%d] Done with page." % self.wid)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	559 self.ctx.work_queue.task_done()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	560 except Exception as ex:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	561 self.ctx.abort_event.set()
39 2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	562 self.abort_exception = ex
2f717f961996 Better error reporting and cache validation. Ludovic Chabant <ludovic@chabant.com> parents: 36 diff changeset	563 logger.debug("[%d] Critical error, aborting." % self.wid)
50 2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	564 if self.ctx.app.debug:
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	565 logger.exception(ex)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	566 break
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	567
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	568 def _unsafeRun(self, job):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	569 start_time = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	570
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	571 bake_res = self._page_baker.bake(job.factory, job.route,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	572 taxonomy_name=job.taxonomy_name,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	573 taxonomy_term=job.taxonomy_term)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	574
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	575 if bake_res.was_baked:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	576 uri = bake_res.out_uris[0]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	577 friendly_uri = uri if uri != '' else '[main page]'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	578 friendly_count = ''
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	579 if bake_res.num_subs > 1:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	580 friendly_count = ' (%d pages)' % bake_res.num_subs
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	581 logger.info(format_timed(start_time, '[%d] %s%s' %
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	582 (self.wid, friendly_uri, friendly_count)))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	583

Mercurial > piecrust2

annotate piecrust/baking/baker.py @ 72:ecee3e8f35e5