Mercurial > piecrust2
changeset 465:b6e797463798
Merge pull request #19 from GitHub.
| author | Ludovic Chabant <ludovic@chabant.com> |
|---|---|
| date | Sat, 11 Jul 2015 20:33:55 -0700 |
| parents | aefd2714d205 (diff) 375301e024b5 (current diff) |
| children | 456db44dcc53 |
| files | |
| diffstat | 85 files changed, 4120 insertions(+), 2268 deletions(-) [+] |
line wrap: on
line diff
--- a/.hgignore Thu Jun 25 05:51:47 2015 +1000 +++ b/.hgignore Sat Jul 11 20:33:55 2015 -0700 @@ -3,8 +3,8 @@ venv tags build/lib -build/messages/_cache -build/messages/_counter +util/messages/_cache +util/messages/_counter dist docs/_cache docs/_counter @@ -13,3 +13,4 @@ piecrust.egg-info piecrust/__version__.py +docs
--- a/.travis.yml Thu Jun 25 05:51:47 2015 +1000 +++ b/.travis.yml Sat Jul 11 20:33:55 2015 -0700 @@ -1,3 +1,4 @@ +sudo: false language: python python: - "3.4"
--- a/build/generate_messages.cmd Thu Jun 25 05:51:47 2015 +1000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,11 +0,0 @@ -@echo off -setlocal - -set CUR_DIR=%~dp0 -set CHEF=%CUR_DIR%..\bin\chef -set OUT_DIR=%CUR_DIR%..\piecrust\resources\messages -set ROOT_DIR=%CUR_DIR%messages - -%CHEF% --root=%ROOT_DIR% bake -o %OUT_DIR% -del %OUT_DIR%\index.html -
--- a/build/generate_messages.sh Thu Jun 25 05:51:47 2015 +1000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,9 +0,0 @@ -#!/bin/sh - -CUR_DIR="$( cd "$( dirname "$0" )" && pwd )" -CHEF=${CUR_DIR}/../bin/chef -OUT_DIR=${CUR_DIR}/../piecrust/resources/messages -ROOT_DIR=${CUR_DIR}/messages - -$CHEF --root=$ROOT_DIR bake -o $OUT_DIR -rm ${OUT_DIR}/index.html
--- a/build/messages/config.yml Thu Jun 25 05:51:47 2015 +1000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,2 +0,0 @@ -site: - title: PieCrust System Messages
--- a/build/messages/pages/_index.html Thu Jun 25 05:51:47 2015 +1000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,12 +0,0 @@ ---- -title: PieCrust System Messages ---- - -Here are the **PieCrust** system message pages: - -* [Requirements Not Met]({{ pcurl('requirements') }}) -* [Error]({{ pcurl('error') }}) -* [Not Found]({{ pcurl('error404') }}) -* [Critical Error]({{ pcurl('critical') }}) - -This very page you're reading, however, is only here for convenience.
--- a/build/messages/pages/critical.html Thu Jun 25 05:51:47 2015 +1000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,6 +0,0 @@ ---- -title: The Whole Kitchen Burned Down! -layout: error ---- -Something critically bad happened, and **PieCrust** needs to shut down. It's probably our fault. -
--- a/build/messages/pages/error.html Thu Jun 25 05:51:47 2015 +1000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,5 +0,0 @@ ---- -title: The Cake Just Burned! -layout: error ---- -
--- a/build/messages/pages/error404.html Thu Jun 25 05:51:47 2015 +1000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,6 +0,0 @@ ---- -title: Can't find the sugar! -layout: error ---- -It looks like the page you were trying to access does not exist around here. Try going somewhere else. -
--- a/build/messages/templates/default.html Thu Jun 25 05:51:47 2015 +1000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,70 +0,0 @@ -<!doctype html> -<html> -<head> - <title>{{ page.title }}</title> - <meta name="generator" content="PieCrust" /> - <link rel="stylesheet" type="text/css" href="http://fonts.googleapis.com/css?family=Lobster"> - <style> - body { - margin: 0; - padding: 1em; - background: #eee; - color: #000; - font-family: Georgia, serif; - } - h1 { - font-size: 4.5em; - font-family: Lobster, 'Trebuchet MS', Verdana, sans-serif; - text-align: center; - font-weight: bold; - margin-top: 0; - color: #333; - text-shadow: 0px 2px 5px rgba(0,0,0,0.3); - } - h2 { - font-size: 2.5em; - font-family: 'Lobster', 'Trebuchet MS', Verdana, sans-serif; - } - code { - background: #ddd; - padding: 0 0.2em; - } - #preamble { - font-size: 1.2em; - font-style: italic; - text-align: center; - margin-bottom: 0; - } - #container { - margin: 0 20%; - } - #content { - margin: 2em 1em; - } - .error-details { - color: #d11; - } - .note { - margin: 3em; - color: #888; - font-style: italic; - } - </style> -</head> -<body> - <div id="container"> - <div id="header"> - <p id="preamble">A Message From The Kitchen:</p> - <h1>{{ page.title }}</h1> - </div> - <hr /> - <div id="content"> - {% block content %} - {{ content|safe }} - {% endblock %} - </div> - <hr /> - {% block footer %}{% endblock %} - </div> -</body> -</html>
--- a/build/messages/templates/error.html Thu Jun 25 05:51:47 2015 +1000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,27 +0,0 @@ -{% extends "default.html" %} - -{% block content %} -{{content|safe}} - -{# The following is `raw` because we want it to be in the - produced page, so it can then be templated on the fly - with the error messages #} -{% raw %} -{% if details %} -<div class="error-details"> - <p>Error details:</p> - <ul> - {% for desc in details %} - <li>{{ desc }}</li> - {% endfor %} - </ul> -</div> -{% endif %} -{% endraw %} -{% endblock %} - -{% block footer %} -{% pcformat 'textile' %} -p(note). You're seeing this because something wrong happend. To see detailed errors with callstacks, run chef with the @--debug@ parameter, append @?!debug@ to the URL, or initialize the @PieCrust@ object with @{'debug': true}@. On the other hand, to see you custom error pages, set the @site/display_errors@ setting to @false@. -{% endpcformat %} -{% endblock %}
--- a/piecrust/__init__.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/__init__.py Sat Jul 11 20:33:55 2015 -0700 @@ -7,6 +7,7 @@ CONFIG_PATH = 'config.yml' THEME_CONFIG_PATH = 'theme_config.yml' THEME_INFO_PATH = 'theme_info.yml' +ASSET_DIR_SUFFIX = '-assets' DEFAULT_FORMAT = 'markdown' DEFAULT_TEMPLATE_ENGINE = 'jinja2'
--- a/piecrust/app.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/app.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,5 +1,6 @@ import re import json +import time import os.path import codecs import hashlib @@ -28,7 +29,7 @@ logger = logging.getLogger(__name__) -CACHE_VERSION = 19 +CACHE_VERSION = 20 class VariantNotFoundError(Exception): @@ -412,10 +413,15 @@ if self.env is None: self.env = StandardEnvironment() self.env.initialize(self) + self.env.registerTimer('SiteConfigLoad') + self.env.registerTimer('PageLoad') + self.env.registerTimer("PageDataBuild") @cached_property def config(self): logger.debug("Creating site configuration...") + start_time = time.perf_counter() + paths = [] if self.theme_dir: paths.append(os.path.join(self.theme_dir, THEME_CONFIG_PATH)) @@ -456,6 +462,7 @@ sc['realm'] = REALM_THEME config.fixups.append(_fixupThemeSources) + self.env.stepTimer('SiteConfigLoad', time.perf_counter() - start_time) return config @cached_property
--- a/piecrust/baking/baker.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/baking/baker.py Sat Jul 11 20:33:55 2015 -0700 @@ -2,12 +2,15 @@ import os.path import hashlib import logging -import threading +import multiprocessing from piecrust.baking.records import ( - TransitionalBakeRecord, BakeRecordPageEntry) -from piecrust.baking.scheduler import BakeScheduler -from piecrust.baking.single import (BakingError, PageBaker) -from piecrust.chefutil import format_timed, log_friendly_exception + BakeRecordEntry, TransitionalBakeRecord, TaxonomyInfo) +from piecrust.baking.worker import ( + save_factory, + JOB_LOAD, JOB_RENDER_FIRST, JOB_BAKE) +from piecrust.chefutil import ( + format_timed_scope, format_timed) +from piecrust.routing import create_route_metadata from piecrust.sources.base import ( REALM_NAMES, REALM_USER, REALM_THEME) @@ -21,7 +24,6 @@ self.app = app self.out_dir = out_dir self.force = force - self.num_workers = app.config.get('baker/workers', 4) # Remember what taxonomy pages we should skip # (we'll bake them repeatedly later with each taxonomy term) @@ -29,17 +31,23 @@ logger.debug("Gathering taxonomy page paths:") for tax in self.app.taxonomies: for src in self.app.sources: - path = tax.resolvePagePath(src.name) - if path is not None: + tax_page_ref = tax.getPageRef(src) + for path in tax_page_ref.possible_paths: self.taxonomy_pages.append(path) logger.debug(" - %s" % path) + # Register some timers. + self.app.env.registerTimer('LoadJob', raise_if_registered=False) + self.app.env.registerTimer('RenderFirstSubJob', + raise_if_registered=False) + self.app.env.registerTimer('BakeJob', raise_if_registered=False) + def bake(self): logger.debug(" Bake Output: %s" % self.out_dir) logger.debug(" Root URL: %s" % self.app.config.get('site/root')) # Get into bake mode. - start_time = time.clock() + start_time = time.perf_counter() self.app.config.set('baker/is_baking', True) self.app.env.base_asset_url_format = '%uri%' @@ -52,35 +60,60 @@ record_cache = self.app.cache.getCache('baker') record_id = hashlib.md5(self.out_dir.encode('utf8')).hexdigest() record_name = record_id + '.record' + previous_record_path = None if not self.force and record_cache.has(record_name): - t = time.clock() - record.loadPrevious(record_cache.getCachePath(record_name)) - logger.debug(format_timed( - t, 'loaded previous bake record', - colored=False)) + with format_timed_scope(logger, "loaded previous bake record", + level=logging.DEBUG, colored=False): + previous_record_path = record_cache.getCachePath(record_name) + record.loadPrevious(previous_record_path) record.current.success = True # Figure out if we need to clean the cache because important things # have changed. - self._handleCacheValidity(record) + is_cache_valid = self._handleCacheValidity(record) + if not is_cache_valid: + previous_record_path = None + + # Pre-create all caches. + for cache_name in ['app', 'baker', 'pages', 'renders']: + self.app.cache.getCache(cache_name) # Gather all sources by realm -- we're going to bake each realm - # separately so we can handle "overlaying" (i.e. one realm overrides - # another realm's pages). + # separately so we can handle "overriding" (i.e. one realm overrides + # another realm's pages, like the user realm overriding the theme + # realm). sources_by_realm = {} for source in self.app.sources: srclist = sources_by_realm.setdefault(source.realm, []) srclist.append(source) + # Create the worker processes. + pool = self._createWorkerPool(previous_record_path) + # Bake the realms. realm_list = [REALM_USER, REALM_THEME] for realm in realm_list: srclist = sources_by_realm.get(realm) if srclist is not None: - self._bakeRealm(record, realm, srclist) + self._bakeRealm(record, pool, realm, srclist) # Bake taxonomies. - self._bakeTaxonomies(record) + self._bakeTaxonomies(record, pool) + + # All done with the workers. Close the pool and get timing reports. + reports = pool.close() + record.current.timers = {} + for i in range(len(reports)): + timers = reports[i] + if timers is None: + continue + + worker_name = 'BakeWorker_%d' % i + record.current.timers[worker_name] = {} + for name, val in timers['data'].items(): + main_val = record.current.timers.setdefault(name, 0) + record.current.timers[name] = main_val + val + record.current.timers[worker_name][name] = val # Delete files from the output. self._handleDeletetions(record) @@ -98,11 +131,11 @@ os.rename(record_path, record_path_next) # Save the bake record. - t = time.clock() - record.current.bake_time = time.time() - record.current.out_dir = self.out_dir - record.saveCurrent(record_cache.getCachePath(record_name)) - logger.debug(format_timed(t, 'saved bake record', colored=False)) + with format_timed_scope(logger, "saved bake record.", + level=logging.DEBUG, colored=False): + record.current.bake_time = time.time() + record.current.out_dir = self.out_dir + record.saveCurrent(record_cache.getCachePath(record_name)) # All done. self.app.config.set('baker/is_baking', False) @@ -111,7 +144,7 @@ return record.detach() def _handleCacheValidity(self, record): - start_time = time.clock() + start_time = time.perf_counter() reason = None if self.force: @@ -146,47 +179,152 @@ logger.info(format_timed( start_time, "cleaned cache (reason: %s)" % reason)) + return False else: record.incremental_count += 1 logger.debug(format_timed( start_time, "cache is assumed valid", colored=False)) + return True - def _bakeRealm(self, record, realm, srclist): - # Gather all page factories from the sources and queue them - # for the workers to pick up. Just skip taxonomy pages for now. - logger.debug("Baking realm %s" % REALM_NAMES[realm]) - pool, queue, abort = self._createWorkerPool(record, self.num_workers) + def _bakeRealm(self, record, pool, realm, srclist): + start_time = time.perf_counter() + try: + record.current.baked_count[realm] = 0 + + all_factories = [] + for source in srclist: + factories = source.getPageFactories() + all_factories += [f for f in factories + if f.path not in self.taxonomy_pages] + + self._loadRealmPages(record, pool, all_factories) + self._renderRealmPages(record, pool, all_factories) + self._bakeRealmPages(record, pool, realm, all_factories) + finally: + page_count = record.current.baked_count[realm] + logger.info(format_timed( + start_time, + "baked %d %s pages." % + (page_count, REALM_NAMES[realm].lower()))) - for source in srclist: - factories = source.getPageFactories() + def _loadRealmPages(self, record, pool, factories): + def _handler(res): + # Create the record entry for this page. + # This will also update the `dirty_source_names` for the record + # as we add page files whose last modification times are later + # than the last bake. + record_entry = BakeRecordEntry(res['source_name'], res['path']) + record_entry.config = res['config'] + if res['errors']: + record_entry.errors += res['errors'] + record.current.success = False + self._logErrors(res['path'], res['errors']) + record.addEntry(record_entry) + + logger.debug("Loading %d realm pages..." % len(factories)) + with format_timed_scope(logger, + "loaded %d pages" % len(factories), + level=logging.DEBUG, colored=False, + timer_env=self.app.env, + timer_category='LoadJob'): + jobs = [] for fac in factories: - if fac.path in self.taxonomy_pages: - logger.debug( - "Skipping taxonomy page: %s:%s" % - (source.name, fac.ref_spec)) + job = { + 'type': JOB_LOAD, + 'job': save_factory(fac)} + jobs.append(job) + ar = pool.queueJobs(jobs, handler=_handler) + ar.wait() + + def _renderRealmPages(self, record, pool, factories): + def _handler(res): + entry = record.getCurrentEntry(res['path']) + if res['errors']: + entry.errors += res['errors'] + record.current.success = False + self._logErrors(res['path'], res['errors']) + + logger.debug("Rendering %d realm pages..." % len(factories)) + with format_timed_scope(logger, + "prepared %d pages" % len(factories), + level=logging.DEBUG, colored=False, + timer_env=self.app.env, + timer_category='RenderFirstSubJob'): + jobs = [] + for fac in factories: + record_entry = record.getCurrentEntry(fac.path) + if record_entry.errors: + logger.debug("Ignoring %s because it had previous " + "errors." % fac.ref_spec) continue - entry = BakeRecordPageEntry(fac.source.name, fac.rel_path, - fac.path) - record.addEntry(entry) + # Make sure the source and the route exist for this page, + # otherwise we add errors to the record entry and we'll skip + # this page for the rest of the bake. + source = self.app.getSource(fac.source.name) + if source is None: + record_entry.errors.append( + "Can't get source for page: %s" % fac.ref_spec) + logger.error(record_entry.errors[-1]) + continue - route = self.app.getRoute(source.name, fac.metadata, + route = self.app.getRoute(fac.source.name, fac.metadata, skip_taxonomies=True) if route is None: - entry.errors.append( + record_entry.errors.append( "Can't get route for page: %s" % fac.ref_spec) - logger.error(entry.errors[-1]) + logger.error(record_entry.errors[-1]) continue - queue.addJob(BakeWorkerJob(fac, route, entry)) + # All good, queue the job. + job = { + 'type': JOB_RENDER_FIRST, + 'job': save_factory(fac)} + jobs.append(job) + + ar = pool.queueJobs(jobs, handler=_handler) + ar.wait() + + def _bakeRealmPages(self, record, pool, realm, factories): + def _handler(res): + entry = record.getCurrentEntry(res['path'], res['taxonomy_info']) + entry.subs = res['sub_entries'] + if res['errors']: + entry.errors += res['errors'] + self._logErrors(res['path'], res['errors']) + if entry.has_any_error: + record.current.success = False + if entry.subs and entry.was_any_sub_baked: + record.current.baked_count[realm] += 1 - success = self._waitOnWorkerPool(pool, abort) - record.current.success &= success + logger.debug("Baking %d realm pages..." % len(factories)) + with format_timed_scope(logger, + "baked %d pages" % len(factories), + level=logging.DEBUG, colored=False, + timer_env=self.app.env, + timer_category='BakeJob'): + jobs = [] + for fac in factories: + job = self._makeBakeJob(record, fac) + if job is not None: + jobs.append(job) - def _bakeTaxonomies(self, record): - logger.debug("Baking taxonomies") + ar = pool.queueJobs(jobs, handler=_handler) + ar.wait() + def _bakeTaxonomies(self, record, pool): + logger.debug("Baking taxonomy pages...") + with format_timed_scope(logger, 'built taxonomy buckets', + level=logging.DEBUG, colored=False): + buckets = self._buildTaxonomyBuckets(record) + + start_time = time.perf_counter() + page_count = self._bakeTaxonomyBuckets(record, pool, buckets) + logger.info(format_timed(start_time, + "baked %d taxonomy pages." % page_count)) + + def _buildTaxonomyBuckets(self, record): # Let's see all the taxonomy terms for which we must bake a # listing page... first, pre-populate our big map of used terms. # For each source name, we have a list of taxonomies, and for each @@ -250,8 +388,19 @@ if not tt_info.dirty_terms.isdisjoint(set(terms)): tt_info.dirty_terms.add(terms) + return buckets + + def _bakeTaxonomyBuckets(self, record, pool, buckets): + def _handler(res): + entry = record.getCurrentEntry(res['path'], res['taxonomy_info']) + entry.subs = res['sub_entries'] + if res['errors']: + entry.errors += res['errors'] + if entry.has_any_error: + record.current.success = False + # Start baking those terms. - pool, queue, abort = self._createWorkerPool(record, self.num_workers) + jobs = [] for source_name, source_taxonomies in buckets.items(): for tax_name, tt_info in source_taxonomies.items(): terms = tt_info.dirty_terms @@ -262,8 +411,8 @@ "Baking '%s' for source '%s': %s" % (tax_name, source_name, terms)) tax = self.app.getTaxonomy(tax_name) - route = self.app.getTaxonomyRoute(tax_name, source_name) - tax_page_ref = tax.getPageRef(source_name) + source = self.app.getSource(source_name) + tax_page_ref = tax.getPageRef(source) if not tax_page_ref.exists: logger.debug( "No taxonomy page found at '%s', skipping." % @@ -273,19 +422,24 @@ logger.debug( "Using taxonomy page: %s:%s" % (tax_page_ref.source_name, tax_page_ref.rel_path)) + fac = tax_page_ref.getFactory() + for term in terms: - fac = tax_page_ref.getFactory() logger.debug( "Queuing: %s [%s=%s]" % (fac.ref_spec, tax_name, term)) - entry = BakeRecordPageEntry( - fac.source.name, fac.rel_path, fac.path, - (tax_name, term, source_name)) - record.addEntry(entry) - queue.addJob(BakeWorkerJob(fac, route, entry)) + tax_info = TaxonomyInfo(tax_name, source_name, term) + + cur_entry = BakeRecordEntry( + fac.source.name, fac.path, tax_info) + record.addEntry(cur_entry) - success = self._waitOnWorkerPool(pool, abort) - record.current.success &= success + job = self._makeBakeJob(record, fac, tax_info) + if job is not None: + jobs.append(job) + + ar = pool.queueJobs(jobs, handler=_handler) + ar.wait() # Now we create bake entries for all the terms that were *not* dirty. # This is because otherwise, on the next incremental bake, we wouldn't @@ -296,18 +450,76 @@ # current version. if (prev_entry and prev_entry.taxonomy_info and not cur_entry): - sn = prev_entry.source_name - tn, tt, tsn = prev_entry.taxonomy_info - tt_info = buckets[tsn][tn] - if tt in tt_info.all_terms: + ti = prev_entry.taxonomy_info + tt_info = buckets[ti.source_name][ti.taxonomy_name] + if ti.term in tt_info.all_terms: logger.debug("Creating unbaked entry for taxonomy " - "term '%s:%s'." % (tn, tt)) + "term '%s:%s'." % (ti.taxonomy_name, ti.term)) record.collapseEntry(prev_entry) else: logger.debug("Taxonomy term '%s:%s' isn't used anymore." % - (tn, tt)) + (ti.taxonomy_name, ti.term)) + + return len(jobs) + + def _makeBakeJob(self, record, fac, tax_info=None): + # Get the previous (if any) and current entry for this page. + pair = record.getPreviousAndCurrentEntries(fac.path, tax_info) + assert pair is not None + prev_entry, cur_entry = pair + assert cur_entry is not None + + # Ignore if there were errors in the previous passes. + if cur_entry.errors: + logger.debug("Ignoring %s because it had previous " + "errors." % fac.ref_spec) + return None + + # Build the route metadata and find the appropriate route. + page = fac.buildPage() + route_metadata = create_route_metadata(page) + if tax_info is not None: + tax = self.app.getTaxonomy(tax_info.taxonomy_name) + route = self.app.getTaxonomyRoute(tax_info.taxonomy_name, + tax_info.source_name) + + slugified_term = route.slugifyTaxonomyTerm(tax_info.term) + route_metadata[tax.term_name] = slugified_term + else: + route = self.app.getRoute(fac.source.name, route_metadata, + skip_taxonomies=True) + assert route is not None + + # Figure out if this page is overriden by another previously + # baked page. This happens for example when the user has + # made a page that has the same page/URL as a theme page. + uri = route.getUri(route_metadata) + override_entry = record.getOverrideEntry(page.path, uri) + if override_entry is not None: + override_source = self.app.getSource( + override_entry.source_name) + if override_source.realm == fac.source.realm: + cur_entry.errors.append( + "Page '%s' maps to URL '%s' but is overriden " + "by page '%s'." % + (fac.ref_spec, uri, override_entry.path)) + logger.error(cur_entry.errors[-1]) + cur_entry.flags |= BakeRecordEntry.FLAG_OVERRIDEN + return None + + job = { + 'type': JOB_BAKE, + 'job': { + 'factory_info': save_factory(fac), + 'taxonomy_info': tax_info, + 'route_metadata': route_metadata, + 'dirty_source_names': record.dirty_source_names + } + } + return job def _handleDeletetions(self, record): + logger.debug("Handling deletions...") for path, reason in record.getDeletions(): logger.debug("Removing '%s': %s" % (path, reason)) try: @@ -318,139 +530,29 @@ # by the user. pass - def _createWorkerPool(self, record, pool_size=4): - pool = [] - queue = BakeScheduler(record) - abort = threading.Event() - for i in range(pool_size): - ctx = BakeWorkerContext( - self.app, self.out_dir, self.force, - record, queue, abort) - worker = BakeWorker(i, ctx) - pool.append(worker) - return pool, queue, abort - - def _waitOnWorkerPool(self, pool, abort): - for w in pool: - w.start() - - success = True - try: - for w in pool: - w.join() - success &= w.success - except KeyboardInterrupt: - logger.warning("Bake aborted by user... " - "waiting for workers to stop.") - abort.set() - for w in pool: - w.join() - raise + def _logErrors(self, path, errors): + rel_path = os.path.relpath(path, self.app.root_dir) + logger.error("Errors found in %s:" % rel_path) + for e in errors: + logger.error(" " + e) - if abort.is_set(): - excs = [w.abort_exception for w in pool - if w.abort_exception is not None] - logger.error("Baking was aborted due to %s error(s):" % len(excs)) - if self.app.debug: - for e in excs: - logger.exception(e) - else: - for e in excs: - log_friendly_exception(logger, e) - raise BakingError("Baking was aborted due to errors.") - - return success - - -class BakeWorkerContext(object): - def __init__(self, app, out_dir, force, record, work_queue, - abort_event): - self.app = app - self.out_dir = out_dir - self.force = force - self.record = record - self.work_queue = work_queue - self.abort_event = abort_event - - -class BakeWorkerJob(object): - def __init__(self, factory, route, record_entry): - self.factory = factory - self.route = route - self.record_entry = record_entry - - @property - def source(self): - return self.factory.source - + def _createWorkerPool(self, previous_record_path): + from piecrust.workerpool import WorkerPool + from piecrust.baking.worker import BakeWorkerContext, BakeWorker -class BakeWorker(threading.Thread): - def __init__(self, wid, ctx): - super(BakeWorker, self).__init__(name=('worker%d' % wid)) - self.wid = wid - self.ctx = ctx - self.abort_exception = None - self.success = True - self._page_baker = PageBaker( - ctx.app, ctx.out_dir, ctx.force, - ctx.record) - - def run(self): - while(not self.ctx.abort_event.is_set()): - try: - job = self.ctx.work_queue.getNextJob(wait_timeout=1) - if job is None: - logger.debug( - "[%d] No more work... shutting down." % - self.wid) - break - success = self._unsafeRun(job) - logger.debug("[%d] Done with page." % self.wid) - self.ctx.work_queue.onJobFinished(job) - self.success &= success - except Exception as ex: - self.ctx.abort_event.set() - self.abort_exception = ex - self.success = False - logger.debug("[%d] Critical error, aborting." % self.wid) - if self.ctx.app.debug: - logger.exception(ex) - break + worker_count = self.app.config.get('baker/workers') + batch_size = self.app.config.get('baker/batch_size') - def _unsafeRun(self, job): - start_time = time.clock() - - entry = job.record_entry - try: - self._page_baker.bake(job.factory, job.route, entry) - except BakingError as ex: - logger.debug("Got baking error. Adding it to the record.") - while ex: - entry.errors.append(str(ex)) - ex = ex.__cause__ - - has_error = False - for e in entry.getAllErrors(): - has_error = True - logger.error(e) - if has_error: - return False - - if entry.was_any_sub_baked: - first_sub = entry.subs[0] - - friendly_uri = first_sub.out_uri - if friendly_uri == '': - friendly_uri = '[main page]' - - friendly_count = '' - if entry.num_subs > 1: - friendly_count = ' (%d pages)' % entry.num_subs - logger.info(format_timed( - start_time, '[%d] %s%s' % - (self.wid, friendly_uri, friendly_count))) - - return True + ctx = BakeWorkerContext( + self.app.root_dir, self.app.cache.base_dir, self.out_dir, + previous_record_path=previous_record_path, + force=self.force, debug=self.app.debug) + pool = WorkerPool( + worker_count=worker_count, + batch_size=batch_size, + worker_class=BakeWorker, + initargs=(ctx,)) + return pool class _TaxonomyTermsInfo(object): @@ -463,3 +565,4 @@ def __repr__(self): return 'dirty:%s, all:%s' % (self.dirty_terms, self.all_terms) +
--- a/piecrust/baking/records.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/baking/records.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,5 +1,6 @@ import copy import os.path +import hashlib import logging from piecrust.records import Record, TransitionalRecord @@ -7,35 +8,31 @@ logger = logging.getLogger(__name__) -def _get_transition_key(source_name, rel_path, taxonomy_info=None): - key = '%s:%s' % (source_name, rel_path) +def _get_transition_key(path, taxonomy_info=None): + key = path if taxonomy_info: - taxonomy_name, taxonomy_term, taxonomy_source_name = taxonomy_info - key += ';%s:%s=' % (taxonomy_source_name, taxonomy_name) - if isinstance(taxonomy_term, tuple): - key += '/'.join(taxonomy_term) + key += '+%s:%s=' % (taxonomy_info.source_name, + taxonomy_info.taxonomy_name) + if isinstance(taxonomy_info.term, tuple): + key += '/'.join(taxonomy_info.term) else: - key += taxonomy_term - return key + key += taxonomy_info.term + return hashlib.md5(key.encode('utf8')).hexdigest() class BakeRecord(Record): - RECORD_VERSION = 12 + RECORD_VERSION = 14 def __init__(self): super(BakeRecord, self).__init__() self.out_dir = None self.bake_time = None + self.baked_count = {} + self.timers = None self.success = True -class BakeRecordPassInfo(object): - def __init__(self): - self.used_source_names = set() - self.used_taxonomy_terms = set() - - -class BakeRecordSubPageEntry(object): +class SubPageBakeInfo(object): FLAG_NONE = 0 FLAG_BAKED = 2**0 FLAG_FORCED_BY_SOURCE = 2**1 @@ -48,7 +45,7 @@ self.out_path = out_path self.flags = self.FLAG_NONE self.errors = [] - self.render_passes = {} + self.render_info = None @property def was_clean(self): @@ -62,13 +59,26 @@ def was_baked_successfully(self): return self.was_baked and len(self.errors) == 0 - def collapseRenderPasses(self, other): - for p, pinfo in self.render_passes.items(): - if p not in other.render_passes: - other.render_passes[p] = copy.deepcopy(pinfo) + def anyPass(self, func): + assert self.render_info is not None + for p, pinfo in self.render_info.items(): + if func(pinfo): + return True + return False + + def copyRenderInfo(self): + assert self.render_info + return copy.deepcopy(self.render_info) -class BakeRecordPageEntry(object): +class TaxonomyInfo(object): + def __init__(self, taxonomy_name, source_name, term): + self.taxonomy_name = taxonomy_name + self.source_name = source_name + self.term = term + + +class BakeRecordEntry(object): """ An entry in the bake record. The `taxonomy_info` attribute should be a tuple of the form: @@ -79,16 +89,14 @@ FLAG_SOURCE_MODIFIED = 2**1 FLAG_OVERRIDEN = 2**2 - def __init__(self, source_name, rel_path, path, taxonomy_info=None): + def __init__(self, source_name, path, taxonomy_info=None): self.source_name = source_name - self.rel_path = rel_path self.path = path self.taxonomy_info = taxonomy_info self.flags = self.FLAG_NONE self.config = None + self.errors = [] self.subs = [] - self.assets = [] - self.errors = [] @property def path_mtime(self): @@ -109,6 +117,20 @@ return True return False + @property + def all_assets(self): + for sub in self.subs: + yield from sub.assets + + @property + def has_any_error(self): + if len(self.errors) > 0: + return True + for o in self.subs: + if len(o.errors) > 0: + return True + return False + def getSub(self, sub_index): return self.subs[sub_index - 1] @@ -120,15 +142,17 @@ def getAllUsedSourceNames(self): res = set() for o in self.subs: - for p, pinfo in o.render_passes.items(): - res |= pinfo.used_source_names + if o.render_info is not None: + for p, pinfo in o.render_info.items(): + res |= pinfo.used_source_names return res def getAllUsedTaxonomyTerms(self): res = set() for o in self.subs: - for p, pinfo in o.render_passes.items(): - res |= pinfo.used_taxonomy_terms + if o.render_info is not None: + for p, pinfo in o.render_info.items(): + res |= pinfo.used_taxonomy_terms return res @@ -141,37 +165,44 @@ def addEntry(self, entry): if (self.previous.bake_time and entry.path_mtime >= self.previous.bake_time): - entry.flags |= BakeRecordPageEntry.FLAG_SOURCE_MODIFIED + entry.flags |= BakeRecordEntry.FLAG_SOURCE_MODIFIED self.dirty_source_names.add(entry.source_name) super(TransitionalBakeRecord, self).addEntry(entry) def getTransitionKey(self, entry): - return _get_transition_key(entry.source_name, entry.rel_path, - entry.taxonomy_info) + return _get_transition_key(entry.path, entry.taxonomy_info) - def getOverrideEntry(self, factory, uri): + def getPreviousAndCurrentEntries(self, path, taxonomy_info=None): + key = _get_transition_key(path, taxonomy_info) + pair = self.transitions.get(key) + return pair + + def getOverrideEntry(self, path, uri): for pair in self.transitions.values(): cur = pair[1] - if (cur and - (cur.source_name != factory.source.name or - cur.rel_path != factory.rel_path)): - for o in cur.subs: - if o.out_uri == uri: - return cur + if cur and cur.path != path: + for o in cur.subs: + if o.out_uri == uri: + return cur return None - def getPreviousEntry(self, source_name, rel_path, taxonomy_info=None): - key = _get_transition_key(source_name, rel_path, taxonomy_info) - pair = self.transitions.get(key) + def getPreviousEntry(self, path, taxonomy_info=None): + pair = self.getPreviousAndCurrentEntries(path, taxonomy_info) if pair is not None: return pair[0] return None + def getCurrentEntry(self, path, taxonomy_info=None): + pair = self.getPreviousAndCurrentEntries(path, taxonomy_info) + if pair is not None: + return pair[1] + return None + def collapseEntry(self, prev_entry): cur_entry = copy.deepcopy(prev_entry) - cur_entry.flags = BakeRecordPageEntry.FLAG_NONE + cur_entry.flags = BakeRecordEntry.FLAG_NONE for o in cur_entry.subs: - o.flags = BakeRecordSubPageEntry.FLAG_NONE + o.flags = SubPageBakeInfo.FLAG_NONE self.addEntry(cur_entry) def getDeletions(self): @@ -187,5 +218,5 @@ yield (p, 'source file changed outputs') def _onNewEntryAdded(self, entry): - entry.flags |= BakeRecordPageEntry.FLAG_NEW + entry.flags |= BakeRecordEntry.FLAG_NEW
--- a/piecrust/baking/single.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/baking/single.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,46 +1,28 @@ import os.path -import copy import shutil import codecs import logging import urllib.parse -from piecrust.baking.records import ( - BakeRecordPassInfo, BakeRecordPageEntry, BakeRecordSubPageEntry) -from piecrust.data.filters import ( - PaginationFilter, HasFilterClause, - IsFilterClause, AndBooleanClause, - page_value_accessor) +from piecrust import ASSET_DIR_SUFFIX +from piecrust.baking.records import SubPageBakeInfo from piecrust.rendering import ( QualifiedPage, PageRenderingContext, render_page, - PASS_FORMATTING, PASS_RENDERING) -from piecrust.sources.base import ( - PageFactory, - REALM_NAMES, REALM_USER, REALM_THEME) + PASS_FORMATTING) from piecrust.uriutil import split_uri logger = logging.getLogger(__name__) -def copy_public_page_config(config): - res = config.getDeepcopy() - for k in list(res.keys()): - if k.startswith('__'): - del res[k] - return res - - class BakingError(Exception): pass class PageBaker(object): - def __init__(self, app, out_dir, force=False, record=None, - copy_assets=True): + def __init__(self, app, out_dir, force=False, copy_assets=True): self.app = app self.out_dir = out_dir self.force = force - self.record = record self.copy_assets = copy_assets self.site_root = app.config.get('site/root') self.pretty_urls = app.config.get('site/pretty_urls') @@ -60,124 +42,41 @@ return os.path.normpath(os.path.join(*bake_path)) - def bake(self, factory, route, record_entry): - # Get the page. - page = factory.buildPage() - route_metadata = copy.deepcopy(factory.metadata) - - # Add taxonomy info in the template data and route metadata if needed. - bake_taxonomy_info = None - if record_entry.taxonomy_info: - tax_name, tax_term, tax_source_name = record_entry.taxonomy_info - taxonomy = self.app.getTaxonomy(tax_name) - slugified_term = route.slugifyTaxonomyTerm(tax_term) - route_metadata[taxonomy.term_name] = slugified_term - bake_taxonomy_info = (taxonomy, tax_term) - - # Generate the URI. - uri = route.getUri(route_metadata, provider=page) - - # See if this URL has been overriden by a previously baked page. - # If that page is from another realm (e.g. a user page vs. a theme - # page), we silently skip this page. If they're from the same realm, - # we don't allow overriding and raise an error (this is probably - # because of a misconfigured configuration that allows for ambiguous - # URLs between 2 routes or sources). - override = self.record.getOverrideEntry(factory, uri) - if override is not None: - override_source = self.app.getSource(override.source_name) - if override_source.realm == factory.source.realm: - raise BakingError( - "Page '%s' maps to URL '%s' but is overriden by page" - "'%s:%s'." % (factory.ref_spec, uri, - override.source_name, - override.rel_path)) - logger.debug("'%s' [%s] is overriden by '%s:%s'. Skipping" % - (factory.ref_spec, uri, override.source_name, - override.rel_path)) - record_entry.flags |= BakeRecordPageEntry.FLAG_OVERRIDEN - return - - # Setup the record entry. - record_entry.config = copy_public_page_config(page.config) - + def bake(self, qualified_page, prev_entry, dirty_source_names, + tax_info=None): # Start baking the sub-pages. cur_sub = 1 has_more_subs = True - force_this = self.force - invalidate_formatting = False - prev_record_entry = self.record.getPreviousEntry( - factory.source.name, factory.rel_path, - record_entry.taxonomy_info) - - logger.debug("Baking '%s'..." % uri) + sub_entries = [] while has_more_subs: # Get the URL and path for this sub-page. - sub_uri = route.getUri(route_metadata, sub_num=cur_sub, - provider=page) + sub_uri = qualified_page.getUri(cur_sub) + logger.debug("Baking '%s' [%d]..." % (sub_uri, cur_sub)) out_path = self.getOutputPath(sub_uri) # Create the sub-entry for the bake record. - record_sub_entry = BakeRecordSubPageEntry(sub_uri, out_path) - record_entry.subs.append(record_sub_entry) + sub_entry = SubPageBakeInfo(sub_uri, out_path) + sub_entries.append(sub_entry) # Find a corresponding sub-entry in the previous bake record. - prev_record_sub_entry = None - if prev_record_entry: + prev_sub_entry = None + if prev_entry: try: - prev_record_sub_entry = prev_record_entry.getSub(cur_sub) + prev_sub_entry = prev_entry.getSub(cur_sub) except IndexError: pass - # Figure out what to do with this page. - if (prev_record_sub_entry and - (prev_record_sub_entry.was_baked_successfully or - prev_record_sub_entry.was_clean)): - # If the current page is known to use pages from other sources, - # see if any of those got baked, or are going to be baked for - # some reason. If so, we need to bake this one too. - # (this happens for instance with the main page of a blog). - dirty_src_names, invalidated_render_passes = ( - self._getDirtySourceNamesAndRenderPasses( - prev_record_sub_entry)) - if len(invalidated_render_passes) > 0: - logger.debug( - "'%s' is known to use sources %s, which have " - "items that got (re)baked. Will force bake this " - "page. " % (uri, dirty_src_names)) - record_sub_entry.flags |= \ - BakeRecordSubPageEntry.FLAG_FORCED_BY_SOURCE - force_this = True - - if PASS_FORMATTING in invalidated_render_passes: - logger.debug( - "Will invalidate cached formatting for '%s' " - "since sources were using during that pass." - % uri) - invalidate_formatting = True - elif (prev_record_sub_entry and - prev_record_sub_entry.errors): - # Previous bake failed. We'll have to bake it again. - logger.debug( - "Previous record entry indicates baking failed for " - "'%s'. Will bake it again." % uri) - record_sub_entry.flags |= \ - BakeRecordSubPageEntry.FLAG_FORCED_BY_PREVIOUS_ERRORS - force_this = True - elif not prev_record_sub_entry: - # No previous record. We'll have to bake it. - logger.debug("No previous record entry found for '%s'. Will " - "force bake it." % uri) - record_sub_entry.flags |= \ - BakeRecordSubPageEntry.FLAG_FORCED_BY_NO_PREVIOUS - force_this = True + # Figure out if we need to invalidate or force anything. + force_this_sub, invalidate_formatting = _compute_force_flags( + prev_sub_entry, sub_entry, dirty_source_names) + force_this_sub = force_this_sub or self.force # Check for up-to-date outputs. do_bake = True - if not force_this: + if not force_this_sub: try: - in_path_time = page.path_mtime + in_path_time = qualified_page.path_mtime out_path_time = os.path.getmtime(out_path) if out_path_time >= in_path_time: do_bake = False @@ -188,10 +87,10 @@ # If this page didn't bake because it's already up-to-date. # Keep trying for as many subs as we know this page has. if not do_bake: - prev_record_sub_entry.collapseRenderPasses(record_sub_entry) - record_sub_entry.flags = BakeRecordSubPageEntry.FLAG_NONE + sub_entry.render_info = prev_sub_entry.copyRenderInfo() + sub_entry.flags = SubPageBakeInfo.FLAG_NONE - if prev_record_entry.num_subs >= cur_sub + 1: + if prev_entry.num_subs >= cur_sub + 1: cur_sub += 1 has_more_subs = True logger.debug(" %s is up to date, skipping to next " @@ -207,34 +106,25 @@ cache_key = sub_uri self.app.env.rendered_segments_repository.invalidate( cache_key) - record_sub_entry.flags |= \ - BakeRecordSubPageEntry.FLAG_FORMATTING_INVALIDATED + sub_entry.flags |= \ + SubPageBakeInfo.FLAG_FORMATTING_INVALIDATED logger.debug(" p%d -> %s" % (cur_sub, out_path)) - qp = QualifiedPage(page, route, route_metadata) - ctx, rp = self._bakeSingle(qp, cur_sub, out_path, - bake_taxonomy_info) + rp = self._bakeSingle(qualified_page, cur_sub, out_path, + tax_info) except Exception as ex: - if self.app.debug: - logger.exception(ex) - page_rel_path = os.path.relpath(page.path, self.app.root_dir) + page_rel_path = os.path.relpath(qualified_page.path, + self.app.root_dir) raise BakingError("%s: error baking '%s'." % - (page_rel_path, uri)) from ex + (page_rel_path, sub_uri)) from ex # Record what we did. - record_sub_entry.flags |= BakeRecordSubPageEntry.FLAG_BAKED - self.record.dirty_source_names.add(record_entry.source_name) - for p, pinfo in ctx.render_passes.items(): - brpi = BakeRecordPassInfo() - brpi.used_source_names = set(pinfo.used_source_names) - brpi.used_taxonomy_terms = set(pinfo.used_taxonomy_terms) - record_sub_entry.render_passes[p] = brpi - if prev_record_sub_entry: - prev_record_sub_entry.collapseRenderPasses(record_sub_entry) + sub_entry.flags |= SubPageBakeInfo.FLAG_BAKED + sub_entry.render_info = rp.copyRenderInfo() # Copy page assets. if (cur_sub == 1 and self.copy_assets and - ctx.used_assets is not None): + sub_entry.anyPass(lambda p: p.used_assets)): if self.pretty_urls: out_assets_dir = os.path.dirname(out_path) else: @@ -244,47 +134,115 @@ out_assets_dir += out_name_noext logger.debug("Copying page assets to: %s" % out_assets_dir) - if not os.path.isdir(out_assets_dir): - os.makedirs(out_assets_dir, 0o755) - for ap in ctx.used_assets: - dest_ap = os.path.join(out_assets_dir, - os.path.basename(ap)) - logger.debug(" %s -> %s" % (ap, dest_ap)) - shutil.copy(ap, dest_ap) - record_entry.assets.append(ap) + _ensure_dir_exists(out_assets_dir) + + page_dirname = os.path.dirname(qualified_page.path) + page_pathname, _ = os.path.splitext(qualified_page.path) + in_assets_dir = page_pathname + ASSET_DIR_SUFFIX + for fn in os.listdir(in_assets_dir): + full_fn = os.path.join(page_dirname, fn) + if os.path.isfile(full_fn): + dest_ap = os.path.join(out_assets_dir, fn) + logger.debug(" %s -> %s" % (full_fn, dest_ap)) + shutil.copy(full_fn, dest_ap) # Figure out if we have more work. has_more_subs = False - if ctx.used_pagination is not None: - if ctx.used_pagination.has_more: - cur_sub += 1 - has_more_subs = True + if sub_entry.anyPass(lambda p: p.pagination_has_more): + cur_sub += 1 + has_more_subs = True + + return sub_entries - def _bakeSingle(self, qualified_page, num, out_path, taxonomy_info=None): + def _bakeSingle(self, qualified_page, num, out_path, tax_info=None): ctx = PageRenderingContext(qualified_page, page_num=num) - if taxonomy_info: - ctx.setTaxonomyFilter(taxonomy_info[0], taxonomy_info[1]) + if tax_info: + tax = self.app.getTaxonomy(tax_info.taxonomy_name) + ctx.setTaxonomyFilter(tax, tax_info.term) rp = render_page(ctx) out_dir = os.path.dirname(out_path) - if not os.path.isdir(out_dir): - os.makedirs(out_dir, 0o755) + _ensure_dir_exists(out_dir) with codecs.open(out_path, 'w', 'utf8') as fp: fp.write(rp.content) - return ctx, rp + return rp + + +def _compute_force_flags(prev_sub_entry, sub_entry, dirty_source_names): + # Figure out what to do with this page. + force_this_sub = False + invalidate_formatting = False + sub_uri = sub_entry.out_uri + if (prev_sub_entry and + (prev_sub_entry.was_baked_successfully or + prev_sub_entry.was_clean)): + # If the current page is known to use pages from other sources, + # see if any of those got baked, or are going to be baked for + # some reason. If so, we need to bake this one too. + # (this happens for instance with the main page of a blog). + dirty_for_this, invalidated_render_passes = ( + _get_dirty_source_names_and_render_passes( + prev_sub_entry, dirty_source_names)) + if len(invalidated_render_passes) > 0: + logger.debug( + "'%s' is known to use sources %s, which have " + "items that got (re)baked. Will force bake this " + "page. " % (sub_uri, dirty_for_this)) + sub_entry.flags |= \ + SubPageBakeInfo.FLAG_FORCED_BY_SOURCE + force_this_sub = True - def _getDirtySourceNamesAndRenderPasses(self, record_sub_entry): - dirty_src_names = set() - invalidated_render_passes = set() - for p, pinfo in record_sub_entry.render_passes.items(): - for src_name in pinfo.used_source_names: - is_dirty = (src_name in self.record.dirty_source_names) - if is_dirty: - invalidated_render_passes.add(p) - dirty_src_names.add(src_name) - break - return dirty_src_names, invalidated_render_passes + if PASS_FORMATTING in invalidated_render_passes: + logger.debug( + "Will invalidate cached formatting for '%s' " + "since sources were using during that pass." + % sub_uri) + invalidate_formatting = True + elif (prev_sub_entry and + prev_sub_entry.errors): + # Previous bake failed. We'll have to bake it again. + logger.debug( + "Previous record entry indicates baking failed for " + "'%s'. Will bake it again." % sub_uri) + sub_entry.flags |= \ + SubPageBakeInfo.FLAG_FORCED_BY_PREVIOUS_ERRORS + force_this_sub = True + elif not prev_sub_entry: + # No previous record. We'll have to bake it. + logger.debug("No previous record entry found for '%s'. Will " + "force bake it." % sub_uri) + sub_entry.flags |= \ + SubPageBakeInfo.FLAG_FORCED_BY_NO_PREVIOUS + force_this_sub = True + + return force_this_sub, invalidate_formatting + +def _get_dirty_source_names_and_render_passes(sub_entry, dirty_source_names): + dirty_for_this = set() + invalidated_render_passes = set() + assert sub_entry.render_info is not None + for p, pinfo in sub_entry.render_info.items(): + for src_name in pinfo.used_source_names: + is_dirty = (src_name in dirty_source_names) + if is_dirty: + invalidated_render_passes.add(p) + dirty_for_this.add(src_name) + break + return dirty_for_this, invalidated_render_passes + + +def _ensure_dir_exists(path): + try: + os.makedirs(path, mode=0o755, exist_ok=True) + except OSError: + # In a multiprocess environment, several process may very + # occasionally try to create the same directory at the same time. + # Let's ignore any error and if something's really wrong (like file + # acces permissions or whatever), then it will more legitimately fail + # just after this when we try to write files. + pass +
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/piecrust/baking/worker.py Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,220 @@ +import time +import logging +from piecrust.app import PieCrust +from piecrust.baking.records import BakeRecord, _get_transition_key +from piecrust.baking.single import PageBaker, BakingError +from piecrust.environment import AbortedSourceUseError +from piecrust.rendering import ( + QualifiedPage, PageRenderingContext, render_page_segments) +from piecrust.routing import create_route_metadata +from piecrust.sources.base import PageFactory +from piecrust.workerpool import IWorker + + +logger = logging.getLogger(__name__) + + +class BakeWorkerContext(object): + def __init__(self, root_dir, sub_cache_dir, out_dir, + previous_record_path=None, + force=False, debug=False): + self.root_dir = root_dir + self.sub_cache_dir = sub_cache_dir + self.out_dir = out_dir + self.previous_record_path = previous_record_path + self.force = force + self.debug = debug + self.app = None + self.previous_record = None + self.previous_record_index = None + + +class BakeWorker(IWorker): + def __init__(self, ctx): + self.ctx = ctx + self.work_start_time = time.perf_counter() + + def initialize(self): + # Create the app local to this worker. + app = PieCrust(self.ctx.root_dir, debug=self.ctx.debug) + app._useSubCacheDir(self.ctx.sub_cache_dir) + app.env.fs_cache_only_for_main_page = True + app.env.registerTimer("BakeWorker_%d_Total" % self.wid) + app.env.registerTimer("BakeWorkerInit") + app.env.registerTimer("JobReceive") + self.ctx.app = app + + # Load previous record + if self.ctx.previous_record_path: + self.ctx.previous_record = BakeRecord.load( + self.ctx.previous_record_path) + self.ctx.previous_record_index = {} + for e in self.ctx.previous_record.entries: + key = _get_transition_key(e.path, e.taxonomy_info) + self.ctx.previous_record_index[key] = e + + # Create the job handlers. + job_handlers = { + JOB_LOAD: LoadJobHandler(self.ctx), + JOB_RENDER_FIRST: RenderFirstSubJobHandler(self.ctx), + JOB_BAKE: BakeJobHandler(self.ctx)} + for jt, jh in job_handlers.items(): + app.env.registerTimer(type(jh).__name__) + self.job_handlers = job_handlers + + app.env.stepTimerSince("BakeWorkerInit", self.work_start_time) + + def process(self, job): + handler = self.job_handlers[job['type']] + with self.ctx.app.env.timerScope(type(handler).__name__): + return handler.handleJob(job['job']) + + def getReport(self): + self.ctx.app.env.stepTimerSince("BakeWorker_%d_Total" % self.wid, + self.work_start_time) + return { + 'type': 'timers', + 'data': self.ctx.app.env._timers} + + +JOB_LOAD, JOB_RENDER_FIRST, JOB_BAKE = range(0, 3) + + +class JobHandler(object): + def __init__(self, ctx): + self.ctx = ctx + + @property + def app(self): + return self.ctx.app + + def handleJob(self, job): + raise NotImplementedError() + + +def _get_errors(ex): + errors = [] + while ex is not None: + errors.append(str(ex)) + ex = ex.__cause__ + return errors + + +def save_factory(fac): + return { + 'source_name': fac.source.name, + 'rel_path': fac.rel_path, + 'metadata': fac.metadata} + + +def load_factory(app, info): + source = app.getSource(info['source_name']) + return PageFactory(source, info['rel_path'], info['metadata']) + + +class LoadJobHandler(JobHandler): + def handleJob(self, job): + # Just make sure the page has been cached. + fac = load_factory(self.app, job) + logger.debug("Loading page: %s" % fac.ref_spec) + result = { + 'source_name': fac.source.name, + 'path': fac.path, + 'config': None, + 'errors': None} + try: + page = fac.buildPage() + page._load() + result['config'] = page.config.getAll() + except Exception as ex: + logger.debug("Got loading error. Sending it to master.") + result['errors'] = _get_errors(ex) + if self.ctx.debug: + logger.exception(ex) + return result + + +class RenderFirstSubJobHandler(JobHandler): + def handleJob(self, job): + # Render the segments for the first sub-page of this page. + fac = load_factory(self.app, job) + + # These things should be OK as they're checked upstream by the baker. + route = self.app.getRoute(fac.source.name, fac.metadata, + skip_taxonomies=True) + assert route is not None + + page = fac.buildPage() + route_metadata = create_route_metadata(page) + qp = QualifiedPage(page, route, route_metadata) + ctx = PageRenderingContext(qp) + self.app.env.abort_source_use = True + + result = { + 'path': fac.path, + 'aborted': False, + 'errors': None} + logger.debug("Preparing page: %s" % fac.ref_spec) + try: + render_page_segments(ctx) + except AbortedSourceUseError: + logger.debug("Page %s was aborted." % fac.ref_spec) + result['aborted'] = True + except Exception as ex: + logger.debug("Got rendering error. Sending it to master.") + result['errors'] = _get_errors(ex) + if self.ctx.debug: + logger.exception(ex) + finally: + self.app.env.abort_source_use = False + return result + + +class BakeJobHandler(JobHandler): + def __init__(self, ctx): + super(BakeJobHandler, self).__init__(ctx) + self.page_baker = PageBaker(ctx.app, ctx.out_dir, ctx.force) + + def handleJob(self, job): + # Actually bake the page and all its sub-pages to the output folder. + fac = load_factory(self.app, job['factory_info']) + + route_metadata = job['route_metadata'] + tax_info = job['taxonomy_info'] + if tax_info is not None: + route = self.app.getTaxonomyRoute(tax_info.taxonomy_name, + tax_info.source_name) + else: + route = self.app.getRoute(fac.source.name, route_metadata, + skip_taxonomies=True) + assert route is not None + + page = fac.buildPage() + qp = QualifiedPage(page, route, route_metadata) + + result = { + 'path': fac.path, + 'taxonomy_info': tax_info, + 'sub_entries': None, + 'errors': None} + dirty_source_names = job['dirty_source_names'] + + previous_entry = None + if self.ctx.previous_record_index is not None: + key = _get_transition_key(fac.path, tax_info) + previous_entry = self.ctx.previous_record_index.get(key) + + logger.debug("Baking page: %s" % fac.ref_spec) + try: + sub_entries = self.page_baker.bake( + qp, previous_entry, dirty_source_names, tax_info) + result['sub_entries'] = sub_entries + + except BakingError as ex: + logger.debug("Got baking error. Sending it to master.") + result['errors'] = _get_errors(ex) + if self.ctx.debug: + logger.exception(ex) + + return result +
--- a/piecrust/cache.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/cache.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,9 +1,12 @@ import os import os.path +import json import shutil import codecs +import hashlib import logging -import threading +import collections +import repoze.lru logger = logging.getLogger(__name__) @@ -12,7 +15,6 @@ class ExtensibleCache(object): def __init__(self, base_dir): self.base_dir = base_dir - self.lock = threading.Lock() self.caches = {} @property @@ -22,15 +24,12 @@ def getCache(self, name): c = self.caches.get(name) if c is None: - with self.lock: - c = self.caches.get(name) - if c is None: - c_dir = os.path.join(self.base_dir, name) - if not os.path.isdir(c_dir): - os.makedirs(c_dir, 0o755) + c_dir = os.path.join(self.base_dir, name) + if not os.path.isdir(c_dir): + os.makedirs(c_dir, 0o755) - c = SimpleCache(c_dir) - self.caches[name] = c + c = SimpleCache(c_dir) + self.caches[name] = c return c def getCacheDir(self, name): @@ -48,6 +47,10 @@ logger.debug("Cleaning cache: %s" % cache_dir) shutil.rmtree(cache_dir) + # Re-create the cache-dir because now our Cache instance points + # to a directory that doesn't exist anymore. + os.makedirs(cache_dir, 0o755) + def clearCaches(self, except_names=None): for name in self.getCacheNames(except_names=except_names): self.clearCache(name) @@ -145,3 +148,70 @@ def clearCaches(self, except_names=None): pass + +def _make_fs_cache_key(key): + return hashlib.md5(key.encode('utf8')).hexdigest() + + +class MemCache(object): + """ Simple memory cache. It can be backed by a simple file-system + cache, but items need to be JSON-serializable to do this. + """ + def __init__(self, size=2048): + self.cache = repoze.lru.LRUCache(size) + self.fs_cache = None + self._last_access_hit = None + self._invalidated_fs_items = set() + + @property + def last_access_hit(self): + return self._last_access_hit + + def invalidate(self, key): + logger.debug("Invalidating cache item '%s'." % key) + self.cache.invalidate(key) + if self.fs_cache: + logger.debug("Invalidating FS cache item '%s'." % key) + fs_key = _make_fs_cache_key(key) + self._invalidated_fs_items.add(fs_key) + + def put(self, key, item, save_to_fs=True): + self.cache.put(key, item) + if self.fs_cache and save_to_fs: + fs_key = _make_fs_cache_key(key) + item_raw = json.dumps(item) + self.fs_cache.write(fs_key, item_raw) + + def get(self, key, item_maker, fs_cache_time=None, save_to_fs=True): + self._last_access_hit = True + item = self.cache.get(key) + if item is None: + if (self.fs_cache is not None and + fs_cache_time is not None): + # Try first from the file-system cache. + fs_key = _make_fs_cache_key(key) + if (fs_key not in self._invalidated_fs_items and + self.fs_cache.isValid(fs_key, fs_cache_time)): + logger.debug("'%s' found in file-system cache." % + key) + item_raw = self.fs_cache.read(fs_key) + item = json.loads( + item_raw, + object_pairs_hook=collections.OrderedDict) + self.cache.put(key, item) + return item + + # Look into the mem-cache. + logger.debug("'%s' not found in cache, must build." % key) + item = item_maker() + self.cache.put(key, item) + self._last_access_hit = False + + # Save to the file-system if needed. + if self.fs_cache is not None and save_to_fs: + item_raw = json.dumps(item) + self.fs_cache.write(fs_key, item_raw) + + return item + +
--- a/piecrust/chefutil.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/chefutil.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,9 +1,21 @@ import time +import logging +import contextlib from colorama import Fore +@contextlib.contextmanager +def format_timed_scope(logger, message, *, level=logging.INFO, colored=True, + timer_env=None, timer_category=None): + start_time = time.perf_counter() + yield + logger.log(level, format_timed(start_time, message, colored=colored)) + if timer_env is not None and timer_category is not None: + timer_env.stepTimer(timer_category, time.perf_counter() - start_time) + + def format_timed(start_time, message, indent_level=0, colored=True): - end_time = time.clock() + end_time = time.perf_counter() indent = indent_level * ' ' time_str = '%8.1f ms' % ((end_time - start_time) * 1000.0) if colored:
--- a/piecrust/commands/builtin/baking.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/commands/builtin/baking.py Sat Jul 11 20:33:55 2015 -0700 @@ -4,17 +4,19 @@ import hashlib import fnmatch import datetime +from colorama import Fore from piecrust.baking.baker import Baker from piecrust.baking.records import ( - BakeRecord, BakeRecordPageEntry, BakeRecordSubPageEntry) + BakeRecord, BakeRecordEntry, SubPageBakeInfo) from piecrust.chefutil import format_timed from piecrust.commands.base import ChefCommand -from piecrust.processing.base import ProcessorPipeline +from piecrust.processing.pipeline import ProcessorPipeline from piecrust.processing.records import ( ProcessorPipelineRecord, - FLAG_PREPARED, FLAG_PROCESSED, FLAG_OVERRIDEN, + FLAG_PREPARED, FLAG_PROCESSED, FLAG_BYPASSED_STRUCTURED_PROCESSING) -from piecrust.rendering import PASS_FORMATTING, PASS_RENDERING +from piecrust.rendering import ( + PASS_FORMATTING, PASS_RENDERING) logger = logging.getLogger(__name__) @@ -36,12 +38,24 @@ help="Force re-baking the entire website.", action='store_true') parser.add_argument( + '-w', '--workers', + help="The number of worker processes to spawn.", + type=int, default=-1) + parser.add_argument( + '--batch-size', + help="The number of jobs per batch.", + type=int, default=-1) + parser.add_argument( '--assets-only', help="Only bake the assets (don't bake the web pages).", action='store_true') parser.add_argument( '--html-only', - help="Only bake HTML files (don't run the asset pipeline).", + help="Only bake the pages (don't run the asset pipeline).", + action='store_true') + parser.add_argument( + '--show-timers', + help="Show detailed timing information.", action='store_true') def run(self, ctx): @@ -49,7 +63,8 @@ os.path.join(ctx.app.root_dir, '_counter')) success = True - start_time = time.clock() + ctx.timers = {} + start_time = time.perf_counter() try: # Bake the site sources. if not ctx.args.assets_only: @@ -59,6 +74,12 @@ if not ctx.args.html_only: success = success & self._bakeAssets(ctx, out_dir) + # Show merged timers. + if ctx.args.show_timers: + logger.info("-------------------") + logger.info("Timing information:") + _show_timers(ctx.timers) + # All done. logger.info('-------------------------') logger.info(format_timed(start_time, 'done baking')) @@ -71,10 +92,15 @@ return 1 def _bakeSources(self, ctx, out_dir): + if ctx.args.workers > 0: + ctx.app.config.set('baker/workers', ctx.args.workers) + if ctx.args.batch_size > 0: + ctx.app.config.set('baker/batch_size', ctx.args.batch_size) baker = Baker( ctx.app, out_dir, force=ctx.args.force) record = baker.bake() + _merge_timers(record.timers, ctx.timers) return record.success def _bakeAssets(self, ctx, out_dir): @@ -82,9 +108,41 @@ ctx.app, out_dir, force=ctx.args.force) record = proc.run() + _merge_timers(record.timers, ctx.timers) return record.success +def _merge_timers(source, target): + if source is None: + return + + for name, val in source.items(): + if isinstance(val, float): + if name not in target: + target[name] = 0 + target[name] += val + elif isinstance(val, dict): + if name not in target: + target[name] = {} + _merge_timers(val, target[name]) + + +def _show_timers(timers, indent=''): + sub_timer_names = [] + for name in sorted(timers.keys()): + if isinstance(timers[name], float): + val_str = '%8.1f s' % timers[name] + logger.info( + "%s[%s%s%s] %s" % + (indent, Fore.GREEN, val_str, Fore.RESET, name)) + else: + sub_timer_names.append(name) + + for name in sub_timer_names: + logger.info('%s:' % name) + _show_timers(timers[name], indent + ' ') + + class ShowRecordCommand(ChefCommand): def __init__(self): super(ShowRecordCommand, self).__init__() @@ -111,6 +169,15 @@ type=int, default=0, help="Show the last Nth bake record.") + parser.add_argument( + '--html-only', + action='store_true', + help="Only show records for pages (not from the asset " + "pipeline).") + parser.add_argument( + '--assets-only', + action='store_true', + help="Only show records for assets (not from pages).") def run(self, ctx): out_dir = ctx.args.output or os.path.join(ctx.app.root_dir, '_counter') @@ -126,12 +193,18 @@ if ctx.args.out: out_pattern = '*%s*' % ctx.args.out.strip('*') + if not ctx.args.assets_only: + self._showBakeRecord(ctx, record_name, pattern, out_pattern) + if not ctx.args.html_only: + self._showProcessingRecord(ctx, record_name, pattern, out_pattern) + + def _showBakeRecord(self, ctx, record_name, pattern, out_pattern): + # Show the bake record. record_cache = ctx.app.cache.getCache('baker') if not record_cache.has(record_name): raise Exception("No record has been created for this output path. " "Did you bake there yet?") - # Show the bake record. record = BakeRecord.load(record_cache.getCachePath(record_name)) logging.info("Bake record for: %s" % record.out_dir) logging.info("From: %s" % record_name) @@ -143,68 +216,90 @@ logging.error("Status: failed") logging.info("Entries:") for entry in record.entries: - if pattern and not fnmatch.fnmatch(entry.rel_path, pattern): + if pattern and not fnmatch.fnmatch(entry.path, pattern): continue if out_pattern and not ( any([o for o in entry.out_paths if fnmatch.fnmatch(o, out_pattern)])): continue - flags = [] - if entry.flags & BakeRecordPageEntry.FLAG_OVERRIDEN: - flags.append('overriden') - - passes = {PASS_RENDERING: 'render', PASS_FORMATTING: 'format'} + flags = _get_flag_descriptions( + entry.flags, + { + BakeRecordEntry.FLAG_NEW: 'new', + BakeRecordEntry.FLAG_SOURCE_MODIFIED: 'modified', + BakeRecordEntry.FLAG_OVERRIDEN: 'overriden'}) logging.info(" - ") - logging.info(" path: %s" % entry.rel_path) - logging.info(" spec: %s:%s" % (entry.source_name, - entry.rel_path)) + + rel_path = os.path.relpath(entry.path, ctx.app.root_dir) + logging.info(" path: %s" % rel_path) + logging.info(" source: %s" % entry.source_name) if entry.taxonomy_info: - tn, t, sn = entry.taxonomy_info - logging.info(" taxonomy: %s (%s:%s)" % - (t, sn, tn)) + ti = entry.taxonomy_info + logging.info(" taxonomy: %s = %s (in %s)" % + (ti.taxonomy_name, ti.term, ti.source_name)) else: logging.info(" taxonomy: <none>") - logging.info(" flags: %s" % ', '.join(flags)) + logging.info(" flags: %s" % _join(flags)) logging.info(" config: %s" % entry.config) + if entry.errors: + logging.error(" errors: %s" % entry.errors) + logging.info(" %d sub-pages:" % len(entry.subs)) for sub in entry.subs: + sub_flags = _get_flag_descriptions( + sub.flags, + { + SubPageBakeInfo.FLAG_BAKED: 'baked', + SubPageBakeInfo.FLAG_FORCED_BY_SOURCE: + 'forced by source', + SubPageBakeInfo.FLAG_FORCED_BY_NO_PREVIOUS: + 'forced by missing previous record entry', + SubPageBakeInfo.FLAG_FORCED_BY_PREVIOUS_ERRORS: + 'forced by previous errors', + SubPageBakeInfo.FLAG_FORMATTING_INVALIDATED: + 'formatting invalidated'}) + logging.info(" - ") logging.info(" URL: %s" % sub.out_uri) - logging.info(" path: %s" % os.path.relpath(sub.out_path, - out_dir)) - logging.info(" baked?: %s" % sub.was_baked) + logging.info(" path: %s" % os.path.relpath( + sub.out_path, record.out_dir)) + logging.info(" flags: %s" % _join(sub_flags)) - sub_flags = [] - if sub.flags & BakeRecordSubPageEntry.FLAG_FORCED_BY_SOURCE: - sub_flags.append('forced by source') - if sub.flags & BakeRecordSubPageEntry.FLAG_FORCED_BY_NO_PREVIOUS: - sub_flags.append('forced by missing previous record entry') - if sub.flags & BakeRecordSubPageEntry.FLAG_FORCED_BY_PREVIOUS_ERRORS: - sub_flags.append('forced by previous errors') - logging.info(" flags: %s" % ', '.join(sub_flags)) - - for p, pi in sub.render_passes.items(): - logging.info(" %s pass:" % passes[p]) - logging.info(" used srcs: %s" % - ', '.join(pi.used_source_names)) - logging.info(" used terms: %s" % - ', '.join( - ['%s (%s:%s)' % (t, sn, tn) - for sn, tn, t in pi.used_taxonomy_terms])) + if sub.render_info: + pass_names = { + PASS_FORMATTING: 'formatting pass', + PASS_RENDERING: 'rendering pass'} + for p, ri in sub.render_info.items(): + logging.info(" - %s" % pass_names[p]) + logging.info(" used sources: %s" % + _join(ri.used_source_names)) + pgn_info = 'no' + if ri.used_pagination: + pgn_info = 'yes' + if ri.pagination_has_more: + pgn_info += ', has more' + logging.info(" used pagination: %s", pgn_info) + logging.info(" used assets: %s", + 'yes' if ri.used_assets else 'no') + logging.info(" used terms: %s" % + _join( + ['%s=%s (%s)' % (tn, t, sn) + for sn, tn, t in + ri.used_taxonomy_terms])) + else: + logging.info(" no render info") if sub.errors: logging.error(" errors: %s" % sub.errors) - logging.info(" assets: %s" % ', '.join(entry.assets)) - if entry.errors: - logging.error(" errors: %s" % entry.errors) - + def _showProcessingRecord(self, ctx, record_name, pattern, out_pattern): record_cache = ctx.app.cache.getCache('proc') if not record_cache.has(record_name): - return + raise Exception("No record has been created for this output path. " + "Did you bake there yet?") # Show the pipeline record. record = ProcessorPipelineRecord.load( @@ -219,37 +314,51 @@ logging.error("Status: failed") logging.info("Entries:") for entry in record.entries: - if pattern and not fnmatch.fnmatch(entry.rel_input, pattern): + rel_path = os.path.relpath(entry.path, ctx.app.root_dir) + if pattern and not fnmatch.fnmatch(rel_path, pattern): continue if out_pattern and not ( any([o for o in entry.rel_outputs if fnmatch.fnmatch(o, out_pattern)])): continue - flags = [] - if entry.flags & FLAG_PREPARED: - flags.append('prepared') - if entry.flags & FLAG_PROCESSED: - flags.append('processed') - if entry.flags & FLAG_OVERRIDEN: - flags.append('overriden') - if entry.flags & FLAG_BYPASSED_STRUCTURED_PROCESSING: - flags.append('external') + flags = _get_flag_descriptions( + entry.flags, + { + FLAG_PREPARED: 'prepared', + FLAG_PROCESSED: 'processed', + FLAG_BYPASSED_STRUCTURED_PROCESSING: 'external'}) + logger.info(" - ") - logger.info(" path: %s" % entry.rel_input) + logger.info(" path: %s" % rel_path) logger.info(" out paths: %s" % entry.rel_outputs) - logger.info(" flags: %s" % flags) - logger.info(" proc tree: %s" % format_proc_tree( + logger.info(" flags: %s" % _join(flags)) + logger.info(" proc tree: %s" % _format_proc_tree( entry.proc_tree, 14*' ')) + if entry.errors: logger.error(" errors: %s" % entry.errors) -def format_proc_tree(tree, margin='', level=0): +def _join(items, sep=', ', text_if_none='none'): + if items: + return sep.join(items) + return text_if_none + + +def _get_flag_descriptions(flags, descriptions): + res = [] + for k, v in descriptions.items(): + if flags & k: + res.append(v) + return res + + +def _format_proc_tree(tree, margin='', level=0): name, children = tree res = '%s%s+ %s\n' % (margin if level > 0 else '', level * ' ', name) if children: for c in children: - res += format_proc_tree(c, margin, level + 1) + res += _format_proc_tree(c, margin, level + 1) return res
--- a/piecrust/configuration.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/configuration.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,65 +1,45 @@ import re -import copy import logging -import collections +import collections.abc import yaml from yaml.constructor import ConstructorError +try: + from yaml import CSafeLoader as SafeLoader +except ImportError: + from yaml import SafeLoader logger = logging.getLogger(__name__) -default_allowed_types = (dict, list, tuple, int, bool, str) +default_allowed_types = (dict, list, tuple, float, int, bool, str) class ConfigurationError(Exception): pass -class Configuration(object): +class Configuration(collections.abc.MutableMapping): def __init__(self, values=None, validate=True): if values is not None: - self.setAll(values, validate) + self.setAll(values, validate=validate) else: self._values = None - def __contains__(self, key): - return self.has(key) - def __getitem__(self, key): - value = self.get(key) - if value is None: - raise KeyError() - return value + self._ensureLoaded() + bits = key.split('/') + cur = self._values + for b in bits: + try: + cur = cur[b] + except KeyError: + raise KeyError("No such item: %s" % key) + return cur def __setitem__(self, key, value): - return self.set(key, value) - - def setAll(self, values, validate=True): - if validate: - self._validateAll(values) - self._values = values - - def getDeepcopy(self, validate_types=False): - if validate_types: - self.validateTypes() - return copy.deepcopy(self.get()) - - def get(self, key_path=None, default_value=None): self._ensureLoaded() - if key_path is None: - return self._values - bits = key_path.split('/') - cur = self._values - for b in bits: - cur = cur.get(b) - if cur is None: - return default_value - return cur - - def set(self, key_path, value): - self._ensureLoaded() - value = self._validateValue(key_path, value) - bits = key_path.split('/') + value = self._validateValue(key, value) + bits = key.split('/') bitslen = len(bits) cur = self._values for i, b in enumerate(bits): @@ -70,15 +50,31 @@ cur[b] = {} cur = cur[b] - def has(self, key_path): + def __delitem__(self, key): + raise NotImplementedError() + + def __iter__(self): + self._ensureLoaded() + return iter(self._values) + + def __len__(self): self._ensureLoaded() - bits = key_path.split('/') - cur = self._values - for b in bits: - cur = cur.get(b) - if cur is None: - return False - return True + return len(self._values) + + def has(self, key): + return key in self + + def set(self, key, value): + self[key] = value + + def setAll(self, values, validate=False): + if validate: + self._validateAll(values) + self._values = values + + def getAll(self): + self._ensureLoaded() + return self._values def merge(self, other): self._ensureLoaded() @@ -174,7 +170,7 @@ return config, offset -class ConfigurationLoader(yaml.SafeLoader): +class ConfigurationLoader(SafeLoader): """ A YAML loader that loads mappings into ordered dictionaries. """ def __init__(self, *args, **kwargs):
--- a/piecrust/data/assetor.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/data/assetor.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,6 +1,7 @@ import os import os.path import logging +from piecrust import ASSET_DIR_SUFFIX from piecrust.uriutil import multi_replace @@ -31,8 +32,6 @@ class Assetor(object): - ASSET_DIR_SUFFIX = '-assets' - debug_render_doc = """Helps render URLs to files in the current page's asset folder.""" debug_render = [] @@ -58,6 +57,10 @@ self._cacheAssets() return map(lambda i: i[0], self._cache.values()) + def _getFilenames(self): + assert self._cache is not None + return map(lambda i: i[1], self._cache.values()) + def _debugRenderAssetNames(self): self._cacheAssets() return list(self._cache.keys()) @@ -68,7 +71,7 @@ self._cache = {} name, ext = os.path.splitext(self._page.path) - assets_dir = name + Assetor.ASSET_DIR_SUFFIX + assets_dir = name + ASSET_DIR_SUFFIX if not os.path.isdir(assets_dir): return @@ -88,6 +91,5 @@ cpi = self._page.app.env.exec_info_stack.current_page_info if cpi is not None: - used_assets = list(map(lambda i: i[1], self._cache.values())) - cpi.render_ctx.used_assets = used_assets + cpi.render_ctx.used_assets = True
--- a/piecrust/data/base.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/data/base.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,217 +1,62 @@ -import copy -import time -import logging -from piecrust.data.assetor import Assetor -from piecrust.uriutil import split_uri - - -logger = logging.getLogger(__name__) - - -class LazyPageConfigLoaderHasNoValue(Exception): - """ An exception that can be returned when a loader for `LazyPageConfig` - can't return any value. - """ - pass +import collections.abc -class LazyPageConfigData(object): - """ An object that represents the configuration header of a page, - but also allows for additional data. It's meant to be exposed - to the templating system. +class MergedMapping(collections.abc.Mapping): + """ Provides a dictionary-like object that's really the aggregation of + multiple dictionary-like objects. """ - debug_render = [] - debug_render_dynamic = ['_debugRenderKeys'] - - def __init__(self, page): - self._page = page - self._values = None - self._loaders = None - - @property - def page(self): - return self._page - - def get(self, name): - try: - return self._getValue(name) - except LazyPageConfigLoaderHasNoValue: - return None + def __init__(self, dicts, path=''): + self._dicts = dicts + self._path = path def __getattr__(self, name): try: - return self._getValue(name) - except LazyPageConfigLoaderHasNoValue: - raise AttributeError + return self[name] + except KeyError: + raise AttributeError("No such attribute: %s" % self._subp(name)) def __getitem__(self, name): - try: - return self._getValue(name) - except LazyPageConfigLoaderHasNoValue: - raise KeyError - - def _getValue(self, name): - self._load() - - if name in self._values: - return self._values[name] - - if self._loaders: - loader = self._loaders.get(name) - if loader is not None: - try: - self._values[name] = loader(self, name) - except LazyPageConfigLoaderHasNoValue: - raise - except Exception as ex: - raise Exception( - "Error while loading attribute '%s' for: %s" % - (name, self._page.rel_path)) from ex - - # We need to double-check `_loaders` here because - # the loader could have removed all loaders, which - # would set this back to `None`. - if self._loaders is not None: - del self._loaders[name] - if len(self._loaders) == 0: - self._loaders = None + values = [] + for d in self._dicts: + try: + val = d[name] + except KeyError: + continue + values.append(val) - else: - loader = self._loaders.get('*') - if loader is not None: - try: - self._values[name] = loader(self, name) - except LazyPageConfigLoaderHasNoValue: - raise - except Exception as ex: - raise Exception( - "Error while loading attribute '%s' for: %s" % - (name, self._page.rel_path)) from ex - # We always keep the wildcard loader in the loaders list. - - if name not in self._values: - raise LazyPageConfigLoaderHasNoValue() - return self._values[name] - - def _setValue(self, name, value): - if self._values is None: - raise Exception("Can't call _setValue before this data has been " - "loaded") - self._values[name] = value + if len(values) == 0: + raise KeyError("No such item: %s" % self._subp(name)) + if len(values) == 1: + return values[0] - def mapLoader(self, attr_name, loader): - if loader is None: - if self._loaders is None or attr_name not in self._loaders: - return - del self._loaders[attr_name] - if len(self._loaders) == 0: - self._loaders = None - return - - if self._loaders is None: - self._loaders = {} - if attr_name in self._loaders: - raise Exception( - "A loader has already been mapped for: %s" % attr_name) - self._loaders[attr_name] = loader - - def _load(self): - if self._values is not None: - return - self._values = self._page.config.getDeepcopy(self._page.app.debug) - try: - self._loadCustom() - except Exception as ex: - raise Exception( - "Error while loading data for: %s" % - self._page.rel_path) from ex - - def _loadCustom(self): - pass + for val in values: + if not isinstance(val, (dict, collections.abc.Mapping)): + raise Exception( + "Template data for '%s' contains an incompatible mix " + "of data: %s" % ( + self._subp(name), + ', '.join([str(type(v)) for v in values]))) - def _debugRenderKeys(self): - self._load() - keys = set(self._values.keys()) - if self._loaders: - keys |= set(self._loaders.keys()) - return list(keys) - - -class PaginationData(LazyPageConfigData): - def __init__(self, page): - super(PaginationData, self).__init__(page) - self._route = None - self._route_metadata = None + return MergedMapping(values, self._subp(name)) - def _get_uri(self): - page = self._page - if self._route is None: - # TODO: this is not quite correct, as we're missing parts of the - # route metadata if the current page is a taxonomy page. - self._route = page.app.getRoute(page.source.name, - page.source_metadata) - self._route_metadata = copy.deepcopy(page.source_metadata) - if self._route is None: - raise Exception("Can't get route for page: %s" % page.path) - return self._route.getUri(self._route_metadata, provider=page) - - def _loadCustom(self): - page_url = self._get_uri() - _, slug = split_uri(self.page.app, page_url) - self._setValue('url', page_url) - self._setValue('slug', slug) - self._setValue( - 'timestamp', - time.mktime(self.page.datetime.timetuple())) - date_format = self.page.app.config.get('site/date_format') - if date_format: - self._setValue('date', self.page.datetime.strftime(date_format)) - - assetor = Assetor(self.page, page_url) - self._setValue('assets', assetor) + def __iter__(self): + keys = set() + for d in self._dicts: + keys |= set(d.keys()) + return iter(keys) - segment_names = self.page.config.get('segments') - for name in segment_names: - self.mapLoader(name, self._load_rendered_segment) - - def _load_rendered_segment(self, data, name): - do_render = True - eis = self._page.app.env.exec_info_stack - if eis is not None and eis.hasPage(self._page): - # This is the pagination data for the page that is currently - # being rendered! Inception! But this is possible... so just - # prevent infinite recursion. - do_render = False - - assert self is data + def __len__(self): + keys = set() + for d in self._dicts: + keys |= set(d.keys()) + return len(keys) - if do_render: - uri = self._get_uri() - try: - from piecrust.rendering import ( - QualifiedPage, PageRenderingContext, - render_page_segments) - qp = QualifiedPage(self._page, self._route, - self._route_metadata) - ctx = PageRenderingContext(qp) - segs = render_page_segments(ctx) - except Exception as e: - raise Exception( - "Error rendering segments for '%s'" % uri) from e - else: - segs = {} - for name in self.page.config.get('segments'): - segs[name] = "<unavailable: current page>" + def _subp(self, name): + return '%s/%s' % (self._path, name) - for k, v in segs.items(): - self.mapLoader(k, None) - self._setValue(k, v) + def _prependMapping(self, d): + self._dicts.insert(0, d) - if 'content.abstract' in segs: - self._setValue('content', segs['content.abstract']) - self._setValue('has_more', True) - if name == 'content': - return segs['content.abstract'] + def _appendMapping(self, d): + self._dicts.append(d) - return segs[name] -
--- a/piecrust/data/builder.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/data/builder.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,14 +1,12 @@ -import re -import time -import copy import logging from werkzeug.utils import cached_property -from piecrust import APP_VERSION -from piecrust.configuration import merge_dicts from piecrust.data.assetor import Assetor -from piecrust.data.debug import build_debug_info +from piecrust.data.base import MergedMapping from piecrust.data.linker import PageLinkerData +from piecrust.data.pagedata import PageData from piecrust.data.paginator import Paginator +from piecrust.data.piecrustdata import PieCrustData +from piecrust.data.providersdata import DataProvidersData from piecrust.uriutil import split_sub_uri @@ -35,9 +33,10 @@ app = ctx.app page = ctx.page first_uri, _ = split_sub_uri(app, ctx.uri) + pgn_source = ctx.pagination_source or get_default_pagination_source(page) pc_data = PieCrustData() - pgn_source = ctx.pagination_source or get_default_pagination_source(page) + config_data = PageData(page, ctx) paginator = Paginator(page, pgn_source, page_num=ctx.page_num, pgn_filter=ctx.pagination_filter) @@ -45,24 +44,17 @@ linker = PageLinkerData(page.source, page.rel_path) data = { 'piecrust': pc_data, - 'page': {}, + 'page': config_data, 'assets': assetor, 'pagination': paginator, 'family': linker } - page_data = data['page'] - page_data.update(copy.deepcopy(page.source_metadata)) - page_data.update(page.config.getDeepcopy(app.debug)) - page_data['url'] = ctx.uri - page_data['timestamp'] = time.mktime(page.datetime.timetuple()) - date_format = app.config.get('site/date_format') - if date_format: - page_data['date'] = page.datetime.strftime(date_format) #TODO: handle slugified taxonomy terms. - site_data = build_site_data(page) - merge_dicts(data, site_data) + site_data = app.config.getAll() + providers_data = DataProvidersData(page) + data = MergedMapping([data, providers_data, site_data]) # Do this at the end because we want all the data to be ready to be # displayed in the debugger window. @@ -78,53 +70,7 @@ if name in page_data: logger.warning("Content segment '%s' will hide existing data." % name) - page_data[name] = txt - - -class PieCrustData(object): - debug_render = ['version', 'url', 'branding', 'debug_info'] - debug_render_invoke = ['version', 'url', 'branding', 'debug_info'] - debug_render_redirect = {'debug_info': '_debugRenderDebugInfo'} - - def __init__(self): - self.version = APP_VERSION - self.url = 'http://bolt80.com/piecrust/' - self.branding = 'Baked with <em><a href="%s">PieCrust</a> %s</em>.' % ( - 'http://bolt80.com/piecrust/', APP_VERSION) - self._page = None - self._data = None - - @property - def debug_info(self): - if self._page is not None and self._data is not None: - return build_debug_info(self._page, self._data) - return '' - - def _enableDebugInfo(self, page, data): - self._page = page - self._data = data - - def _debugRenderDebugInfo(self): - return "The very thing you're looking at!" - - -re_endpoint_sep = re.compile(r'[\/\.]') - - -def build_site_data(page): - app = page.app - data = app.config.getDeepcopy(app.debug) - for source in app.sources: - endpoint_bits = re_endpoint_sep.split(source.data_endpoint) - endpoint = data - for e in endpoint_bits[:-1]: - if e not in endpoint: - endpoint[e] = {} - endpoint = endpoint[e] - user_data = endpoint.get(endpoint_bits[-1]) - provider = source.buildDataProvider(page, user_data) - endpoint[endpoint_bits[-1]] = provider - return data + page_data._prependMapping(contents) def get_default_pagination_source(page):
--- a/piecrust/data/debug.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/data/debug.py Sat Jul 11 20:33:55 2015 -0700 @@ -3,6 +3,7 @@ import html import logging import collections +import collections.abc from piecrust import APP_VERSION, PIECRUST_URL from piecrust.page import FLAG_RAW_CACHE_VALID @@ -162,7 +163,7 @@ self._write('<null>') return - if isinstance(data, dict): + if isinstance(data, (dict, collections.abc.Mapping)): self._renderCollapsableValueStart(path) with IndentScope(self): self._renderDict(data, path) @@ -208,7 +209,8 @@ self._renderDoc(data, path) self._renderAttributes(data, path) rendered_count = self._renderIterable(data, path, lambda d: enumerate(d)) - if rendered_count == 0: + if (rendered_count == 0 and + not hasattr(data.__class__, 'debug_render_not_empty')): self._writeLine('<p style="%s %s">(empty array)</p>' % (CSS_P, CSS_DOC)) self._writeLine('</div>') @@ -218,7 +220,8 @@ self._renderAttributes(data, path) rendered_count = self._renderIterable(data, path, lambda d: sorted(iter(d.items()), key=lambda i: i[0])) - if rendered_count == 0: + if (rendered_count == 0 and + not hasattr(data.__class__, 'debug_render_not_empty')): self._writeLine('<p style="%s %s">(empty dictionary)</p>' % (CSS_P, CSS_DOC)) self._writeLine('</div>') @@ -240,10 +243,12 @@ data.__class__.debug_render_items): rendered_count = self._renderIterable(data, path, lambda d: enumerate(d)) - if rendered_count == 0: + if (rendered_count == 0 and + not hasattr(data.__class__, 'debug_render_not_empty')): self._writeLine('<p style="%s %s">(empty)</p>' % (CSS_P, CSS_DOC)) - elif rendered_attrs == 0: + elif (rendered_attrs == 0 and + not hasattr(data.__class__, 'debug_render_not_empty')): self._writeLine('<p style="%s %s">(empty)</p>' % (CSS_P, CSS_DOC)) self._writeLine('</div>') @@ -265,6 +270,13 @@ self._writeLine('<span style="%s">– %s</span>' % (CSS_DOC, data.__class__.debug_render_doc)) + if hasattr(data.__class__, 'debug_render_doc_dynamic'): + drdd = data.__class__.debug_render_doc_dynamic + for ng in drdd: + doc = getattr(data, ng) + self._writeLine('<span style="%s">– %s</span>' % + (CSS_DOC, doc())) + doc = self.external_docs.get(path) if doc is not None: self._writeLine('<span style="%s">– %s</span>' %
--- a/piecrust/data/iterators.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/data/iterators.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,6 +1,8 @@ import logging from piecrust.data.filters import PaginationFilter +from piecrust.environment import AbortedSourceUseError from piecrust.events import Event +from piecrust.sources.base import PageSource from piecrust.sources.interfaces import IPaginationSource @@ -108,6 +110,10 @@ class PageIterator(object): + debug_render = [] + debug_render_doc_dynamic = ['_debugRenderDoc'] + debug_render_not_empty = True + def __init__(self, source, current_page=None, pagination_filter=None, offset=0, limit=-1, locked=False): self._source = source @@ -285,6 +291,13 @@ if self._pagesData is not None: return + if (self._current_page is not None and + self._current_page.app.env.abort_source_use and + isinstance(self._source, PageSource)): + logger.debug("Aborting iteration from %s." % + self._current_page.ref_spec) + raise AbortedSourceUseError() + self._ensureSorter() it_chain = self._pages @@ -303,3 +316,6 @@ pn_it = self._source.getTailIterator(iter(pn)) self._prev_page, self._next_page = (list(pn_it)) + def _debugRenderDoc(self): + return "Contains %d items" % len(self) +
--- a/piecrust/data/linker.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/data/linker.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,7 +1,8 @@ import logging import collections -from piecrust.data.base import PaginationData, LazyPageConfigLoaderHasNoValue from piecrust.data.iterators import PageIterator +from piecrust.data.pagedata import LazyPageConfigLoaderHasNoValue +from piecrust.data.paginationdata import PaginationData from piecrust.sources.interfaces import IPaginationSource, IListableSource @@ -12,15 +13,22 @@ """ Entry template data to get access to related pages from a given root page. """ + debug_render = ['parent', 'ancestors', 'siblings', 'children', 'root', + 'forpath'] + debug_render_invoke = ['parent', 'siblings', 'children'] + def __init__(self, source, page_path): self._source = source self._root_page_path = page_path self._linker = None + self._is_loaded = False @property def parent(self): self._load() - return self._linker.parent + if self._linker is not None: + return self._linker.parent + return None @property def ancestors(self): @@ -32,11 +40,15 @@ @property def siblings(self): self._load() + if self._linker is None: + return [] return self._linker @property def children(self): self._load() + if self._linker is None: + return [] self._linker._load() if self._linker._self_item is None: return [] @@ -48,14 +60,24 @@ @property def root(self): self._load() + if self._linker is None: + return None return self._linker.root def forpath(self, rel_path): self._load() + if self._linker is None: + return None return self._linker.forpath(rel_path) def _load(self): - if self._linker is not None: + if self._is_loaded: + return + + self._is_loaded = True + + is_listable = isinstance(self._source, IListableSource) + if not is_listable: return dir_path = self._source.getDirpath(self._root_page_path) @@ -69,7 +91,10 @@ `Paginator` and other page iterators, but with a few additional data like hierarchical data. """ - debug_render = ['is_dir', 'is_self'] + PaginationData.debug_render + debug_render = (['is_dir', 'is_self', 'parent', 'children'] + + PaginationData.debug_render) + debug_render_invoke = (['is_dir', 'is_self', 'parent', 'children'] + + PaginationData.debug_render_invoke) def __init__(self, page): super(LinkedPageData, self).__init__(page) @@ -79,7 +104,7 @@ self.is_page = True self._child_linker = page._linker_info.child_linker - self.mapLoader('*', self._linkerChildLoader) + self._mapLoader('*', self._linkerChildLoader) @property def parent(self): @@ -269,36 +294,35 @@ items = list(self._source.listPath(self._dir_path)) self._items = collections.OrderedDict() - with self._source.app.env.page_repository.startBatchGet(): - for is_dir, name, data in items: - # If `is_dir` is true, `data` will be the directory's source - # path. If not, it will be a page factory. - if is_dir: - item = Linker(self._source, data, - root_page_path=self._root_page_path) - else: - page = data.buildPage() - is_self = (page.rel_path == self._root_page_path) - item = _LinkedPage(page) - item._linker_info.name = name - item._linker_info.is_self = is_self - if is_self: - self._self_item = item + for is_dir, name, data in items: + # If `is_dir` is true, `data` will be the directory's source + # path. If not, it will be a page factory. + if is_dir: + item = Linker(self._source, data, + root_page_path=self._root_page_path) + else: + page = data.buildPage() + is_self = (page.rel_path == self._root_page_path) + item = _LinkedPage(page) + item._linker_info.name = name + item._linker_info.is_self = is_self + if is_self: + self._self_item = item - existing = self._items.get(name) - if existing is None: - self._items[name] = item - elif is_dir: - # The current item is a directory. The existing item - # should be a page. - existing._linker_info.child_linker = item - existing._linker_info.is_dir = True - else: - # The current item is a page. The existing item should - # be a directory. - item._linker_info.child_linker = existing - item._linker_info.is_dir = True - self._items[name] = item + existing = self._items.get(name) + if existing is None: + self._items[name] = item + elif is_dir: + # The current item is a directory. The existing item + # should be a page. + existing._linker_info.child_linker = item + existing._linker_info.is_dir = True + else: + # The current item is a page. The existing item should + # be a directory. + item._linker_info.child_linker = existing + item._linker_info.is_dir = True + self._items[name] = item def filter_page_items(item):
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/piecrust/data/pagedata.py Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,158 @@ +import time +import collections.abc + + +class LazyPageConfigLoaderHasNoValue(Exception): + """ An exception that can be returned when a loader for `LazyPageConfig` + can't return any value. + """ + pass + + +class LazyPageConfigData(collections.abc.Mapping): + """ An object that represents the configuration header of a page, + but also allows for additional data. It's meant to be exposed + to the templating system. + """ + debug_render = [] + debug_render_invoke = [] + debug_render_dynamic = ['_debugRenderKeys'] + debug_render_invoke_dynamic = ['_debugRenderKeys'] + + def __init__(self, page): + self._page = page + self._values = {} + self._loaders = {} + self._is_loaded = False + + def __getattr__(self, name): + try: + return self._getValue(name) + except LazyPageConfigLoaderHasNoValue as ex: + raise AttributeError("No such attribute: %s" % name) from ex + + def __getitem__(self, name): + try: + return self._getValue(name) + except LazyPageConfigLoaderHasNoValue as ex: + raise KeyError("No such key: %s" % name) from ex + + def __iter__(self): + keys = list(self._page.config.keys()) + keys += list(self._values.keys()) + keys += list(self._loaders.keys()) + return iter(keys) + + def __len__(self): + return len(self._page.config) + len(self._values) + len(self._loaders) + + def _getValue(self, name): + # First try the page configuration itself. + try: + return self._page.config[name] + except KeyError: + pass + + # Then try loaded values. + self._ensureLoaded() + try: + return self._values[name] + except KeyError: + pass + + # Try a loader for a new value. + loader = self._loaders.get(name) + if loader is not None: + try: + self._values[name] = loader(self, name) + except LazyPageConfigLoaderHasNoValue: + raise + except Exception as ex: + raise Exception( + "Error while loading attribute '%s' for: %s" % + (name, self._page.rel_path)) from ex + + # Forget this loader now that it served its purpose. + try: + del self._loaders[name] + except KeyError: + pass + return self._values[name] + + # Try the wildcard loader if it exists. + loader = self._loaders.get('*') + if loader is not None: + try: + self._values[name] = loader(self, name) + except LazyPageConfigLoaderHasNoValue: + raise + except Exception as ex: + raise Exception( + "Error while loading attribute '%s' for: %s" % + (name, self._page.rel_path)) from ex + # We always keep the wildcard loader in the loaders list. + return self._values[name] + + raise LazyPageConfigLoaderHasNoValue() + + def _setValue(self, name, value): + self._values[name] = value + + def _unmapLoader(self, attr_name): + try: + del self._loaders[attr_name] + except KeyError: + pass + + def _mapLoader(self, attr_name, loader, override_existing=False): + assert loader is not None + + if not override_existing and attr_name in self._loaders: + raise Exception( + "A loader has already been mapped for: %s" % attr_name) + self._loaders[attr_name] = loader + + def _mapValue(self, attr_name, value, override_existing=False): + loader = lambda _, __: value + self._mapLoader(attr_name, loader, override_existing=override_existing) + + def _ensureLoaded(self): + if self._is_loaded: + return + + self._is_loaded = True + try: + self._load() + except Exception as ex: + raise Exception( + "Error while loading data for: %s" % + self._page.rel_path) from ex + + def _load(self): + pass + + def _debugRenderKeys(self): + self._ensureLoaded() + keys = set(self._values.keys()) + if self._loaders: + keys |= set(self._loaders.keys()) + return list(keys) + + +class PageData(LazyPageConfigData): + """ Template data for a page. + """ + def __init__(self, page, ctx): + super(PageData, self).__init__(page) + self._ctx = ctx + + def _load(self): + page = self._page + for k, v in page.source_metadata.items(): + self._setValue(k, v) + self._setValue('url', self._ctx.uri) + self._setValue('timestamp', time.mktime(page.datetime.timetuple())) + date_format = page.app.config.get('site/date_format') + if date_format: + self._setValue('date', page.datetime.strftime(date_format)) +
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/piecrust/data/paginationdata.py Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,88 @@ +import time +from piecrust.data.assetor import Assetor +from piecrust.data.pagedata import LazyPageConfigData +from piecrust.routing import create_route_metadata +from piecrust.uriutil import split_uri + + +class PaginationData(LazyPageConfigData): + def __init__(self, page): + super(PaginationData, self).__init__(page) + self._route = None + self._route_metadata = None + + def _get_uri(self): + page = self._page + if self._route is None: + # TODO: this is not quite correct, as we're missing parts of the + # route metadata if the current page is a taxonomy page. + route_metadata = create_route_metadata(page) + self._route = page.app.getRoute(page.source.name, route_metadata) + self._route_metadata = route_metadata + if self._route is None: + raise Exception("Can't get route for page: %s" % page.path) + return self._route.getUri(self._route_metadata) + + def _load(self): + page = self._page + page_url = self._get_uri() + _, slug = split_uri(page.app, page_url) + self._setValue('url', page_url) + self._setValue('slug', slug) + self._setValue( + 'timestamp', + time.mktime(page.datetime.timetuple())) + date_format = page.app.config.get('site/date_format') + if date_format: + self._setValue('date', page.datetime.strftime(date_format)) + self._setValue('mtime', page.path_mtime) + + assetor = Assetor(page, page_url) + self._setValue('assets', assetor) + + segment_names = page.config.get('segments') + for name in segment_names: + self._mapLoader(name, self._load_rendered_segment) + + def _load_rendered_segment(self, data, name): + do_render = True + eis = self._page.app.env.exec_info_stack + if eis is not None and eis.hasPage(self._page): + # This is the pagination data for the page that is currently + # being rendered! Inception! But this is possible... so just + # prevent infinite recursion. + do_render = False + + assert self is data + + if do_render: + uri = self._get_uri() + try: + from piecrust.rendering import ( + QualifiedPage, PageRenderingContext, + render_page_segments) + qp = QualifiedPage(self._page, self._route, + self._route_metadata) + ctx = PageRenderingContext(qp) + render_result = render_page_segments(ctx) + segs = render_result.segments + except Exception as e: + raise Exception( + "Error rendering segments for '%s'" % uri) from e + else: + segs = {} + for name in self._page.config.get('segments'): + segs[name] = "<unavailable: current page>" + + for k, v in segs.items(): + self._unmapLoader(k) + self._setValue(k, v) + + if 'content.abstract' in segs: + self._setValue('content', segs['content.abstract']) + self._setValue('has_more', True) + if name == 'content': + return segs['content.abstract'] + + return segs[name] +
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/piecrust/data/piecrustdata.py Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,38 @@ +import logging +from piecrust import APP_VERSION +from piecrust.data.debug import build_debug_info + + +logger = logging.getLogger(__name__) + + +class PieCrustData(object): + debug_render = ['version', 'url', 'branding', 'debug_info'] + debug_render_invoke = ['version', 'url', 'branding', 'debug_info'] + debug_render_redirect = {'debug_info': '_debugRenderDebugInfo'} + + def __init__(self): + self.version = APP_VERSION + self.url = 'http://bolt80.com/piecrust/' + self.branding = 'Baked with <em><a href="%s">PieCrust</a> %s</em>.' % ( + 'http://bolt80.com/piecrust/', APP_VERSION) + self._page = None + self._data = None + + @property + def debug_info(self): + if self._page is not None and self._data is not None: + try: + return build_debug_info(self._page, self._data) + except Exception as ex: + logger.exception(ex) + return ('An error occured while generating debug info. ' + 'Please check the logs.') + return '' + + def _enableDebugInfo(self, page, data): + self._page = page + self._data = data + + def _debugRenderDebugInfo(self): + return "The very thing you're looking at!"
--- a/piecrust/data/provider.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/data/provider.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,54 +1,37 @@ import time +import collections.abc from piecrust.data.iterators import PageIterator from piecrust.sources.array import ArraySource class DataProvider(object): - debug_render_dynamic = ['_debugRenderUserData'] - debug_render_invoke_dynamic = ['_debugRenderUserData'] + debug_render_dynamic = [] + debug_render_invoke_dynamic = [] - def __init__(self, source, page, user_data): + def __init__(self, source, page, override): if source.app is not page.app: raise Exception("The given source and page don't belong to " "the same application.") self._source = source self._page = page - self._user_data = user_data - - def __getattr__(self, name): - if self._user_data is not None: - try: - return self._user_data[name] - except KeyError: - pass - raise AttributeError() - - def __getitem__(self, name): - if self._user_data is not None: - return self._user_data[name] - raise KeyError() - - def _debugRenderUserData(self): - if self._user_data: - return list(self._user_data.keys()) - return [] class IteratorDataProvider(DataProvider): PROVIDER_NAME = 'iterator' - debug_render_doc = """Provides a list of pages.""" + debug_render_doc_dynamic = ['_debugRenderDoc'] + debug_render_not_empty = True - def __init__(self, source, page, user_data): + def __init__(self, source, page, override): + super(IteratorDataProvider, self).__init__(source, page, override) + self._innerIt = None - if isinstance(user_data, IteratorDataProvider): + if isinstance(override, IteratorDataProvider): # Iterator providers can be chained, like for instance with # `site.pages` listing both the theme pages and the user site's # pages. - self._innerIt = user_data - user_data = None + self._innerIt = override - super(IteratorDataProvider, self).__init__(source, page, user_data) self._pages = PageIterator(source, current_page=page) self._pages._iter_event += self._onIteration self._ctx_set = False @@ -70,61 +53,68 @@ eis.current_page_info.render_ctx.addUsedSource(self._source.name) self._ctx_set = True + def _debugRenderDoc(self): + return 'Provides a list of %d items' % len(self) -class BlogDataProvider(DataProvider): + +class BlogDataProvider(DataProvider, collections.abc.Mapping): PROVIDER_NAME = 'blog' debug_render_doc = """Provides a list of blog posts and yearly/monthly archives.""" - debug_render = ['posts', 'years', 'months'] debug_render_dynamic = (['_debugRenderTaxonomies'] + DataProvider.debug_render_dynamic) - def __init__(self, source, page, user_data): - super(BlogDataProvider, self).__init__(source, page, user_data) + def __init__(self, source, page, override): + super(BlogDataProvider, self).__init__(source, page, override) self._yearly = None self._monthly = None self._taxonomies = {} self._ctx_set = False - def __getattr__(self, name): - if self._source.app.getTaxonomy(name) is not None: + def __getitem__(self, name): + if name == 'posts': + return self._posts() + elif name == 'years': + return self._buildYearlyArchive() + elif name == 'months': + return self._buildMonthlyArchive() + elif self._source.app.getTaxonomy(name) is not None: return self._buildTaxonomy(name) - return super(BlogDataProvider, self).__getattr__(name) + raise KeyError("No such item: %s" % name) + + def __iter__(self): + keys = ['posts', 'years', 'months'] + keys += [t.name for t in self._source.app.taxonomies] + return iter(keys) - @property - def posts(self): + def __len__(self): + return 3 + len(self._source.app.taxonomies) + + def _debugRenderTaxonomies(self): + return [t.name for t in self._source.app.taxonomies] + + def _posts(self): it = PageIterator(self._source, current_page=self._page) it._iter_event += self._onIteration return it - @property - def years(self): - return self._buildYearlyArchive() - - @property - def months(self): - return self._buildMonthlyArchive() - - def _debugRenderTaxonomies(self): - return [t.name for t in self._source.app.taxonomies] - def _buildYearlyArchive(self): if self._yearly is not None: return self._yearly self._yearly = [] + yearly_index = {} for post in self._source.getPages(): year = post.datetime.strftime('%Y') - posts_this_year = next( - filter(lambda y: y.name == year, self._yearly), - None) + posts_this_year = yearly_index.get(year) if posts_this_year is None: timestamp = time.mktime( (post.datetime.year, 1, 1, 0, 0, 0, 0, 0, -1)) posts_this_year = BlogArchiveEntry(self._page, year, timestamp) self._yearly.append(posts_this_year) + yearly_index[year] = posts_this_year posts_this_year._data_source.append(post) self._yearly = sorted(self._yearly,
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/piecrust/data/providersdata.py Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,39 @@ +import re +import collections.abc + + +re_endpoint_sep = re.compile(r'[\/\.]') + + +class DataProvidersData(collections.abc.Mapping): + def __init__(self, page): + self._page = page + self._dict = None + + def __getitem__(self, name): + self._load() + return self._dict[name] + + def __iter__(self): + self._load() + return iter(self._dict) + + def __len__(self): + self._load() + return len(self._dict) + + def _load(self): + if self._dict is not None: + return + + self._dict = {} + for source in self._page.app.sources: + endpoint_bits = re_endpoint_sep.split(source.data_endpoint) + endpoint = self._dict + for e in endpoint_bits[:-1]: + if e not in endpoint: + endpoint[e] = {} + endpoint = endpoint[e] + override = endpoint.get(endpoint_bits[-1]) + provider = source.buildDataProvider(self._page, override) + endpoint[endpoint_bits[-1]] = provider
--- a/piecrust/environment.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/environment.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,83 +1,14 @@ -import re import time -import json import logging -import hashlib -import threading import contextlib -import collections -import repoze.lru +from piecrust.cache import MemCache logger = logging.getLogger(__name__) -re_fs_cache_key = re.compile(r'[^\d\w\-\._]+') - - -def _make_fs_cache_key(key): - return hashlib.md5(key.encode('utf8')).hexdigest() - - -class MemCache(object): - """ Simple memory cache. It can be backed by a simple file-system - cache, but items need to be JSON-serializable to do this. - """ - def __init__(self, size=2048): - self.cache = repoze.lru.LRUCache(size) - self.fs_cache = None - self._invalidated_fs_items = set() - self._lock = threading.RLock() - - @contextlib.contextmanager - def startBatchGet(self): - logger.debug("Starting batch cache operation.") - with self._lock: - yield - logger.debug("Ending batch cache operation.") - - def invalidate(self, key): - with self._lock: - logger.debug("Invalidating cache item '%s'." % key) - self.cache.invalidate(key) - if self.fs_cache: - logger.debug("Invalidating FS cache item '%s'." % key) - fs_key = _make_fs_cache_key(key) - self._invalidated_fs_items.add(fs_key) - - def get(self, key, item_maker, fs_cache_time=None): - item = self.cache.get(key) - if item is None: - logger.debug("Acquiring lock for: %s" % key) - with self._lock: - item = self.cache.get(key) - if item is None: - if (self.fs_cache is not None and - fs_cache_time is not None): - # Try first from the file-system cache. - fs_key = _make_fs_cache_key(key) - if (fs_key not in self._invalidated_fs_items and - self.fs_cache.isValid(fs_key, fs_cache_time)): - logger.debug("'%s' found in file-system cache." % - key) - item_raw = self.fs_cache.read(fs_key) - item = json.loads( - item_raw, - object_pairs_hook=collections.OrderedDict) - self.cache.put(key, item) - return item - - # Look into the mem-cache. - logger.debug("'%s' not found in cache, must build." % key) - item = item_maker() - self.cache.put(key, item) - - # Save to the file-system if needed. - if (self.fs_cache is not None and - fs_cache_time is not None): - item_raw = json.dumps(item) - self.fs_cache.write(fs_key, item_raw) - return item +class AbortedSourceUseError(Exception): + pass class ExecutionInfo(object): @@ -85,10 +16,10 @@ self.page = page self.render_ctx = render_ctx self.was_cache_valid = False - self.start_time = time.clock() + self.start_time = time.perf_counter() -class ExecutionInfoStack(threading.local): +class ExecutionInfoStack(object): def __init__(self): self._page_stack = [] @@ -123,6 +54,7 @@ class Environment(object): def __init__(self): + self.app = None self.start_time = None self.exec_info_stack = ExecutionInfoStack() self.was_cache_cleaned = False @@ -131,14 +63,52 @@ self.rendered_segments_repository = MemCache() self.fs_caches = { 'renders': self.rendered_segments_repository} + self.fs_cache_only_for_main_page = False + self.abort_source_use = False + self._default_layout_extensions = None + self._timers = {} + + @property + def default_layout_extensions(self): + if self._default_layout_extensions is not None: + return self._default_layout_extensions + + if self.app is None: + raise Exception("This environment has not been initialized yet.") + + from piecrust.rendering import get_template_engine + dte = get_template_engine(self.app, None) + self._default_layout_extensions = ['.' + e.lstrip('.') + for e in dte.EXTENSIONS] + return self._default_layout_extensions def initialize(self, app): - self.start_time = time.clock() + self.app = app + self.start_time = time.perf_counter() self.exec_info_stack.clear() self.was_cache_cleaned = False self.base_asset_url_format = '%uri%' + self._onSubCacheDirChanged(app) + def registerTimer(self, category, *, raise_if_registered=True): + if raise_if_registered and category in self._timers: + raise Exception("Timer '%s' has already been registered." % + category) + self._timers[category] = 0 + + @contextlib.contextmanager + def timerScope(self, category): + start = time.perf_counter() + yield + self._timers[category] += time.perf_counter() - start + + def stepTimer(self, category, value): + self._timers[category] += value + + def stepTimerSince(self, category, since): + self.stepTimer(category, time.perf_counter() - since) + def _onSubCacheDirChanged(self, app): for name, repo in self.fs_caches.items(): cache = app.cache.getCache(name)
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/piecrust/fastpickle.py Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,221 @@ +import sys +import json +import datetime +import collections + + +def pickle(obj): + data = _pickle_object(obj) + data = json.dumps(data, indent=None, separators=(',', ':')) + return data.encode('utf8') + + +def unpickle(data): + data = data.decode('utf8') + data = json.loads(data) + return _unpickle_object(data) + + +_PICKLING = 0 +_UNPICKLING = 1 + +_identity_dispatch = object() + + +def _tuple_convert(obj, func, op): + if op == _PICKLING: + return ['__type__:tuple'] + [func(c) for c in obj] + elif op == _UNPICKLING: + return tuple([func(c) for c in obj[1:]]) + + +def _list_convert(obj, func, op): + return [func(c) for c in obj] + + +def _dict_convert(obj, func, op): + res = {} + for k, v in obj.items(): + res[k] = func(v) + return res + + +def _ordered_dict_convert(obj, func, op): + if op == _PICKLING: + res = {'__type__': 'OrderedDict'} + for k, v in obj.items(): + res[k] = func(v) + return res + elif op == _UNPICKLING: + res = collections.OrderedDict() + for k, v in obj.items(): + res[k] = func(v) + return res + + +def _set_convert(obj, func, op): + if op == _PICKLING: + return ['__type__:set'] + [func(c) for c in obj] + elif op == _UNPICKLING: + return set([func(c) for c in obj[1:]]) + + +def _date_convert(obj, func, op): + if op == _PICKLING: + return {'__class__': 'date', + 'year': obj.year, + 'month': obj.month, + 'day': obj.day} + elif op == _UNPICKLING: + return datetime.date( + obj['year'], obj['month'], obj['day']) + + +def _datetime_convert(obj, func, op): + if op == _PICKLING: + return {'__class__': 'datetime', + 'year': obj.year, + 'month': obj.month, + 'day': obj.day, + 'hour': obj.hour, + 'minute': obj.minute, + 'second': obj.second, + 'microsecond': obj.microsecond} + elif op == _UNPICKLING: + return datetime.datetime( + obj['year'], obj['month'], obj['day'], + obj['hour'], obj['minute'], obj['second'], obj['microsecond']) + + +def _time_convert(obj, func, op): + if op == _PICKLING: + return {'__class__': 'time', + 'hour': obj.hour, + 'minute': obj.minute, + 'second': obj.second, + 'microsecond': obj.microsecond} + elif op == _UNPICKLING: + return datetime.time( + obj['hour'], obj['minute'], obj['second'], obj['microsecond']) + + +_type_convert = { + type(None): _identity_dispatch, + bool: _identity_dispatch, + int: _identity_dispatch, + float: _identity_dispatch, + str: _identity_dispatch, + datetime.date: _date_convert, + datetime.datetime: _datetime_convert, + datetime.time: _time_convert, + tuple: _tuple_convert, + list: _list_convert, + dict: _dict_convert, + set: _set_convert, + collections.OrderedDict: _ordered_dict_convert, + } + + +_type_unconvert = { + type(None): _identity_dispatch, + bool: _identity_dispatch, + int: _identity_dispatch, + float: _identity_dispatch, + str: _identity_dispatch, + 'date': _date_convert, + 'datetime': _datetime_convert, + 'time': _time_convert, + } + + +_collection_unconvert = { + '__type__:tuple': _tuple_convert, + '__type__:set': _set_convert, + } + + +_mapping_unconvert = { + 'OrderedDict': _ordered_dict_convert + } + + +def _pickle_object(obj): + t = type(obj) + conv = _type_convert.get(t) + + # Object doesn't need conversion? + if conv is _identity_dispatch: + return obj + + # Object has special conversion? + if conv is not None: + return conv(obj, _pickle_object, _PICKLING) + + # Use instance dictionary, or a custom state. + getter = getattr(obj, '__getstate__', None) + if getter is not None: + state = getter() + else: + state = obj.__dict__ + + state = _dict_convert(state, _pickle_object, _PICKLING) + state['__class__'] = obj.__class__.__name__ + state['__module__'] = obj.__class__.__module__ + + return state + + +def _unpickle_object(state): + t = type(state) + conv = _type_unconvert.get(t) + + # Object doesn't need conversion? + if conv is _identity_dispatch: + return state + + # Try collection or mapping conversion. + if t is list: + try: + col_type = state[0] + if not isinstance(col_type, str): + col_type = None + except IndexError: + col_type = None + if col_type is not None: + conv = _collection_unconvert.get(col_type) + if conv is not None: + return conv(state, _unpickle_object, _UNPICKLING) + return _list_convert(state, _unpickle_object, _UNPICKLING) + + assert t is dict + + # Custom mapping type? + map_type = state.get('__type__') + if map_type: + conv = _mapping_unconvert.get(map_type) + return conv(state, _unpickle_object, _UNPICKLING) + + # Class instance or other custom type. + class_name = state.get('__class__') + if class_name is None: + return _dict_convert(state, _unpickle_object, _UNPICKLING) + + conv = _type_unconvert.get(class_name) + if conv is not None: + return conv(state, _unpickle_object, _UNPICKLING) + + mod_name = state['__module__'] + mod = sys.modules[mod_name] + class_def = getattr(mod, class_name) + obj = class_def.__new__(class_def) + + del state['__class__'] + del state['__module__'] + attr_names = list(state.keys()) + for name in attr_names: + state[name] = _unpickle_object(state[name]) + + obj.__dict__.update(state) + + return obj +
--- a/piecrust/formatting/base.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/formatting/base.py Sat Jul 11 20:33:55 2015 -0700 @@ -10,6 +10,7 @@ def __init__(self): self.priority = PRIORITY_NORMAL + self.enabled = True def initialize(self, app): self.app = app
--- a/piecrust/formatting/markdownformatter.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/formatting/markdownformatter.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,4 +1,4 @@ -from markdown import markdown +from markdown import Markdown from piecrust.formatting.base import Formatter @@ -8,15 +8,15 @@ def __init__(self): super(MarkdownFormatter, self).__init__() - self._extensions = None + self._formatter = None def render(self, format_name, txt): assert format_name in self.FORMAT_NAMES self._ensureInitialized() - return markdown(txt, extensions=self._extensions) + return self._formatter.reset().convert(txt) def _ensureInitialized(self): - if self._extensions is not None: + if self._formatter is not None: return config = self.app.config.get('markdown') @@ -34,5 +34,6 @@ # Compatibility with PieCrust 1.x if config.get('use_markdown_extra'): extensions.append('extra') - self._extensions = extensions + self._formatter = Markdown(extensions=extensions) +
--- a/piecrust/formatting/textileformatter.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/formatting/textileformatter.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,4 +1,3 @@ -from textile import textile from piecrust.formatting.base import Formatter @@ -7,6 +6,7 @@ OUTPUT_FORMAT = 'html' def render(self, format_name, text): + from textile import textile assert format_name in self.FORMAT_NAMES return textile(text)
--- a/piecrust/page.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/page.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,7 +1,6 @@ import re import sys import json -import codecs import os.path import hashlib import logging @@ -9,7 +8,8 @@ import dateutil.parser import collections from werkzeug.utils import cached_property -from piecrust.configuration import (Configuration, ConfigurationError, +from piecrust.configuration import ( + Configuration, ConfigurationError, parse_config_header) from piecrust.routing import IRouteMetadataProvider @@ -198,19 +198,18 @@ class ContentSegment(object): debug_render_func = 'debug_render' - def __init__(self, content=None, fmt=None): + def __init__(self): self.parts = [] - if content is not None: - self.parts.append(ContentSegmentPart(content, fmt)) def debug_render(self): return '\n'.join([p.content for p in self.parts]) class ContentSegmentPart(object): - def __init__(self, content, fmt=None, line=-1): + def __init__(self, content, fmt=None, offset=-1, line=-1): self.content = content self.fmt = fmt + self.offset = offset self.line = line def __str__(self): @@ -222,7 +221,8 @@ for key, seg_data in data.items(): seg = ContentSegment() for p_data in seg_data: - part = ContentSegmentPart(p_data['c'], p_data['f'], p_data['l']) + part = ContentSegmentPart(p_data['c'], p_data['f'], p_data['o'], + p_data['l']) seg.parts.append(part) segments[key] = seg return segments @@ -233,7 +233,8 @@ for key, seg in segments.items(): seg_data = [] for part in seg.parts: - p_data = {'c': part.content, 'f': part.fmt, 'l': part.line} + p_data = {'c': part.content, 'f': part.fmt, 'o': part.offset, + 'l': part.line} seg_data.append(p_data) data[key] = seg_data return data @@ -241,9 +242,11 @@ def load_page(app, path, path_mtime=None): try: - return _do_load_page(app, path, path_mtime) + with app.env.timerScope('PageLoad'): + return _do_load_page(app, path, path_mtime) except Exception as e: - logger.exception("Error loading page: %s" % + logger.exception( + "Error loading page: %s" % os.path.relpath(path, app.root_dir)) _, __, traceback = sys.exc_info() raise PageLoadingError(path, e).with_traceback(traceback) @@ -255,20 +258,22 @@ cache_path = hashlib.md5(path.encode('utf8')).hexdigest() + '.json' page_time = path_mtime or os.path.getmtime(path) if cache.isValid(cache_path, page_time): - cache_data = json.loads(cache.read(cache_path), + cache_data = json.loads( + cache.read(cache_path), object_pairs_hook=collections.OrderedDict) - config = PageConfiguration(values=cache_data['config'], + config = PageConfiguration( + values=cache_data['config'], validate=False) content = json_load_segments(cache_data['content']) return config, content, True # Nope, load the page from the source file. logger.debug("Loading page configuration from: %s" % path) - with codecs.open(path, 'r', 'utf-8') as fp: + with open(path, 'r', encoding='utf-8') as fp: raw = fp.read() header, offset = parse_config_header(raw) - if not 'format' in header: + if 'format' not in header: auto_formats = app.config.get('site/auto_formats') name, ext = os.path.splitext(path) header['format'] = auto_formats.get(ext, None) @@ -279,7 +284,7 @@ # Save to the cache. cache_data = { - 'config': config.get(), + 'config': config.getAll(), 'content': json_save_segments(content)} cache.write(cache_path, json.dumps(cache_data)) @@ -352,7 +357,8 @@ # First part, before the first format change. part_text = raw[start:matches[0].start()] parts.append( - ContentSegmentPart(part_text, first_part_fmt, line_offset)) + ContentSegmentPart(part_text, first_part_fmt, start, + line_offset)) line_offset += _count_lines(part_text) for i in range(1, num_matches): @@ -361,17 +367,20 @@ part_text = raw[m1.end() + 1:m2.start()] parts.append( ContentSegmentPart( - part_text, m1.group('fmt'), line_offset)) + part_text, m1.group('fmt'), m1.end() + 1, + line_offset)) line_offset += _count_lines(part_text) lastm = matches[-1] part_text = raw[lastm.end() + 1:end] parts.append(ContentSegmentPart( - part_text, lastm.group('fmt'), line_offset)) + part_text, lastm.group('fmt'), lastm.end() + 1, + line_offset)) return parts, line_offset else: part_text = raw[start:end] - parts = [ContentSegmentPart(part_text, first_part_fmt, line_offset)] + parts = [ContentSegmentPart(part_text, first_part_fmt, start, + line_offset)] return parts, line_offset
--- a/piecrust/plugins/base.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/plugins/base.py Sat Jul 11 20:33:55 2015 -0700 @@ -50,17 +50,23 @@ def getFormatters(self): return self._getPluginComponents( - 'getFormatters', True, order_key=lambda f: f.priority) + 'getFormatters', + initialize=True, register_timer=True, + order_key=lambda f: f.priority) def getTemplateEngines(self): - return self._getPluginComponents('getTemplateEngines', True) + return self._getPluginComponents( + 'getTemplateEngines', + initialize=True, register_timer=True) def getDataProviders(self): return self._getPluginComponents('getDataProviders') def getProcessors(self): return self._getPluginComponents( - 'getProcessors', True, order_key=lambda p: p.priority) + 'getProcessors', + initialize=True, register_timer=True, + order_key=lambda p: p.priority) def getImporters(self): return self._getPluginComponents('getImporters') @@ -86,8 +92,9 @@ to_install = self.app.config.get('site/plugins') if to_install: - for p in to_install: - self._loadPlugin(p) + for name in to_install: + plugin = self._loadPlugin(name) + self._plugins.append(plugin) for plugin in self._plugins: plugin.initialize(self.app) @@ -113,9 +120,11 @@ (plugin_name, ex)) return - self._plugins.append(plugin) + return plugin - def _getPluginComponents(self, name, initialize=False, order_key=None): + def _getPluginComponents(self, name, *, + initialize=False, register_timer=False, + order_key=None): if name in self._componentCache: return self._componentCache[name] @@ -123,10 +132,15 @@ for plugin in self.plugins: plugin_components = getattr(plugin, name)() all_components += plugin_components + if initialize: for comp in plugin_components: comp.initialize(self.app) + if register_timer: + for comp in plugin_components: + self.app.env.registerTimer(comp.__class__.__name__) + if order_key is not None: all_components.sort(key=order_key)
--- a/piecrust/processing/base.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/processing/base.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,35 +1,36 @@ -import re -import time import shutil import os.path import logging -import hashlib -import threading -from queue import Queue, Empty -from piecrust.chefutil import format_timed -from piecrust.processing.records import ( - ProcessorPipelineRecordEntry, TransitionalProcessorPipelineRecord, - FLAG_PREPARED, FLAG_PROCESSED, FLAG_OVERRIDEN, - FLAG_BYPASSED_STRUCTURED_PROCESSING) -from piecrust.processing.tree import ( - ProcessingTreeBuilder, ProcessingTreeRunner, - ProcessingTreeError, ProcessorError, - STATE_DIRTY, - print_node, get_node_name_tree) logger = logging.getLogger(__name__) -re_ansicolors = re.compile('\033\\[\d+m') - - PRIORITY_FIRST = -1 PRIORITY_NORMAL = 0 PRIORITY_LAST = 1 -split_processor_names_re = re.compile(r'[ ,]+') +class PipelineContext(object): + def __init__(self, worker_id, app, out_dir, tmp_dir, force=None): + self.worker_id = worker_id + self.app = app + self.out_dir = out_dir + self.tmp_dir = tmp_dir + self.force = force + self.record = None + self._additional_ignore_patterns = [] + + @property + def is_first_worker(self): + return self.worker_id == 0 + + @property + def is_pipeline_process(self): + return self.worker_id < 0 + + def addIgnorePatterns(self, patterns): + self._additional_ignore_patterns += patterns class Processor(object): @@ -43,10 +44,10 @@ def initialize(self, app): self.app = app - def onPipelineStart(self, pipeline): + def onPipelineStart(self, ctx): pass - def onPipelineEnd(self, pipeline): + def onPipelineEnd(self, ctx): pass def matches(self, path): @@ -117,341 +118,3 @@ return self.stderr_data -class ProcessingContext(object): - def __init__(self, base_dir, mount_info, job_queue, record=None): - self.base_dir = base_dir - self.mount_info = mount_info - self.job_queue = job_queue - self.record = record - - -class ProcessorPipeline(object): - def __init__(self, app, out_dir, force=False): - assert app and out_dir - self.app = app - self.out_dir = out_dir - self.force = force - - tmp_dir = app.sub_cache_dir - if not tmp_dir: - import tempfile - tmp_dir = os.path.join(tempfile.gettempdir(), 'piecrust') - self.tmp_dir = os.path.join(tmp_dir, 'proc') - - baker_params = app.config.get('baker') or {} - - assets_dirs = baker_params.get('assets_dirs', app.assets_dirs) - self.mounts = make_mount_infos(assets_dirs, self.app.root_dir) - - self.num_workers = baker_params.get('workers', 4) - - ignores = baker_params.get('ignore', []) - ignores += [ - '_cache', '_counter', - 'theme_info.yml', - '.DS_Store', 'Thumbs.db', - '.git*', '.hg*', '.svn'] - self.skip_patterns = make_re(ignores) - self.force_patterns = make_re(baker_params.get('force', [])) - - self.processors = app.plugin_loader.getProcessors() - - def addSkipPatterns(self, patterns): - self.skip_patterns += make_re(patterns) - - def filterProcessors(self, authorized_names): - self.processors = self.getFilteredProcessors(authorized_names) - - def getFilteredProcessors(self, authorized_names): - if not authorized_names or authorized_names == 'all': - return self.processors - - if isinstance(authorized_names, str): - authorized_names = split_processor_names_re.split(authorized_names) - - procs = [] - has_star = 'all' in authorized_names - for p in self.processors: - for name in authorized_names: - if name == p.PROCESSOR_NAME: - procs.append(p) - break - if name == ('-%s' % p.PROCESSOR_NAME): - break - else: - if has_star: - procs.append(p) - return procs - - def run(self, src_dir_or_file=None, *, - delete=True, previous_record=None, save_record=True): - # Invoke pre-processors. - for proc in self.processors: - proc.onPipelineStart(self) - - # Sort our processors again in case the pre-process step involved - # patching the processors with some new ones. - self.processors.sort(key=lambda p: p.priority) - - # Create the pipeline record. - record = TransitionalProcessorPipelineRecord() - record_cache = self.app.cache.getCache('proc') - record_name = ( - hashlib.md5(self.out_dir.encode('utf8')).hexdigest() + - '.record') - if previous_record: - record.setPrevious(previous_record) - elif not self.force and record_cache.has(record_name): - t = time.clock() - record.loadPrevious(record_cache.getCachePath(record_name)) - logger.debug(format_timed(t, 'loaded previous bake record', - colored=False)) - logger.debug("Got %d entries in process record." % - len(record.previous.entries)) - - # Create the workers. - pool = [] - queue = Queue() - abort = threading.Event() - pipeline_lock = threading.Lock() - for i in range(self.num_workers): - ctx = ProcessingWorkerContext(self, record, - queue, abort, pipeline_lock) - worker = ProcessingWorker(i, ctx) - worker.start() - pool.append(worker) - - if src_dir_or_file is not None: - # Process only the given path. - # Find out what mount point this is in. - for name, info in self.mounts.items(): - path = info['path'] - if src_dir_or_file[:len(path)] == path: - base_dir = path - mount_info = info - break - else: - known_roots = [i['path'] for i in self.mounts.values()] - raise Exception("Input path '%s' is not part of any known " - "mount point: %s" % - (src_dir_or_file, known_roots)) - - ctx = ProcessingContext(base_dir, mount_info, queue, record) - logger.debug("Initiating processing pipeline on: %s" % src_dir_or_file) - if os.path.isdir(src_dir_or_file): - self.processDirectory(ctx, src_dir_or_file) - elif os.path.isfile(src_dir_or_file): - self.processFile(ctx, src_dir_or_file) - - else: - # Process everything. - for name, info in self.mounts.items(): - path = info['path'] - ctx = ProcessingContext(path, info, queue, record) - logger.debug("Initiating processing pipeline on: %s" % path) - self.processDirectory(ctx, path) - - # Wait on all workers. - record.current.success = True - for w in pool: - w.join() - record.current.success &= w.success - if abort.is_set(): - raise Exception("Worker pool was aborted.") - - # Handle deletions. - if delete: - for path, reason in record.getDeletions(): - logger.debug("Removing '%s': %s" % (path, reason)) - try: - os.remove(path) - except FileNotFoundError: - pass - logger.info('[delete] %s' % path) - - # Invoke post-processors. - for proc in self.processors: - proc.onPipelineEnd(self) - - # Finalize the process record. - record.current.process_time = time.time() - record.current.out_dir = self.out_dir - record.collapseRecords() - - # Save the process record. - if save_record: - t = time.clock() - record.saveCurrent(record_cache.getCachePath(record_name)) - logger.debug(format_timed(t, 'saved bake record', colored=False)) - - return record.detach() - - def processDirectory(self, ctx, start_dir): - for dirpath, dirnames, filenames in os.walk(start_dir): - rel_dirpath = os.path.relpath(dirpath, start_dir) - dirnames[:] = [d for d in dirnames - if not re_matchany(d, self.skip_patterns, rel_dirpath)] - - for filename in filenames: - if re_matchany(filename, self.skip_patterns, rel_dirpath): - continue - self.processFile(ctx, os.path.join(dirpath, filename)) - - def processFile(self, ctx, path): - logger.debug("Queuing: %s" % path) - job = ProcessingWorkerJob(ctx.base_dir, ctx.mount_info, path) - ctx.job_queue.put_nowait(job) - - -class ProcessingWorkerContext(object): - def __init__(self, pipeline, record, - work_queue, abort_event, pipeline_lock): - self.pipeline = pipeline - self.record = record - self.work_queue = work_queue - self.abort_event = abort_event - self.pipeline_lock = pipeline_lock - - -class ProcessingWorkerJob(object): - def __init__(self, base_dir, mount_info, path): - self.base_dir = base_dir - self.mount_info = mount_info - self.path = path - - -class ProcessingWorker(threading.Thread): - def __init__(self, wid, ctx): - super(ProcessingWorker, self).__init__() - self.wid = wid - self.ctx = ctx - self.success = True - - def run(self): - while(not self.ctx.abort_event.is_set()): - try: - job = self.ctx.work_queue.get(True, 0.1) - except Empty: - logger.debug("[%d] No more work... shutting down." % self.wid) - break - - try: - success = self._unsafeRun(job) - logger.debug("[%d] Done with file." % self.wid) - self.ctx.work_queue.task_done() - self.success &= success - except Exception as ex: - self.ctx.abort_event.set() - self.success = False - logger.error("[%d] Critical error, aborting." % self.wid) - logger.exception(ex) - break - - def _unsafeRun(self, job): - start_time = time.clock() - pipeline = self.ctx.pipeline - record = self.ctx.record - - rel_path = os.path.relpath(job.path, job.base_dir) - previous_entry = record.getPreviousEntry(rel_path) - - record_entry = ProcessorPipelineRecordEntry(job.base_dir, rel_path) - record.addEntry(record_entry) - - # Figure out if a previously processed file is overriding this one. - # This can happen if a theme file (processed via a mount point) - # is overridden in the user's website. - if record.current.hasOverrideEntry(rel_path): - record_entry.flags |= FLAG_OVERRIDEN - logger.info(format_timed(start_time, - '%s [not baked, overridden]' % rel_path)) - return True - - processors = pipeline.getFilteredProcessors( - job.mount_info['processors']) - try: - builder = ProcessingTreeBuilder(processors) - tree_root = builder.build(rel_path) - record_entry.flags |= FLAG_PREPARED - except ProcessingTreeError as ex: - msg = str(ex) - logger.error("Error preparing %s:\n%s" % (rel_path, msg)) - while ex: - record_entry.errors.append(str(ex)) - ex = ex.__cause__ - return False - - print_node(tree_root, recursive=True) - leaves = tree_root.getLeaves() - record_entry.rel_outputs = [l.path for l in leaves] - record_entry.proc_tree = get_node_name_tree(tree_root) - if tree_root.getProcessor().is_bypassing_structured_processing: - record_entry.flags |= FLAG_BYPASSED_STRUCTURED_PROCESSING - - force = (pipeline.force or previous_entry is None or - not previous_entry.was_processed_successfully) - if not force: - force = re_matchany(rel_path, pipeline.force_patterns) - - if force: - tree_root.setState(STATE_DIRTY, True) - - try: - runner = ProcessingTreeRunner( - job.base_dir, pipeline.tmp_dir, - pipeline.out_dir, self.ctx.pipeline_lock) - if runner.processSubTree(tree_root): - record_entry.flags |= FLAG_PROCESSED - logger.info(format_timed( - start_time, "[%d] %s" % (self.wid, rel_path))) - return True - except ProcessingTreeError as ex: - msg = str(ex) - if isinstance(ex, ProcessorError): - msg = str(ex.__cause__) - logger.error("Error processing %s:\n%s" % (rel_path, msg)) - while ex: - msg = re_ansicolors.sub('', str(ex)) - record_entry.errors.append(msg) - ex = ex.__cause__ - return False - - -def make_mount_infos(mounts, root_dir): - if isinstance(mounts, list): - mounts = {m: {} for m in mounts} - - for name, info in mounts.items(): - if not isinstance(info, dict): - raise Exception("Asset directory info for '%s' is not a " - "dictionary." % name) - info.setdefault('processors', 'all -uglifyjs -cleancss') - info['path'] = os.path.join(root_dir, name) - - return mounts - - -def make_re(patterns): - re_patterns = [] - for pat in patterns: - if pat[0] == '/' and pat[-1] == '/' and len(pat) > 2: - re_patterns.append(pat[1:-1]) - else: - escaped_pat = (re.escape(pat) - .replace(r'\*', r'[^/\\]*') - .replace(r'\?', r'[^/\\]')) - re_patterns.append(escaped_pat) - return [re.compile(p) for p in re_patterns] - - -def re_matchany(filename, patterns, dirname=None): - if dirname and dirname != '.': - filename = os.path.join(dirname, filename) - - # skip patterns use a forward slash regardless of the platform. - filename = filename.replace('\\', '/') - for pattern in patterns: - if pattern.search(filename): - return True - return False -
--- a/piecrust/processing/compass.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/processing/compass.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,7 +1,6 @@ import os import os.path import logging -import platform import subprocess from piecrust.processing.base import Processor, PRIORITY_FIRST from piecrust.uriutil import multi_replace
--- a/piecrust/processing/less.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/processing/less.py Sat Jul 11 20:33:55 2015 -0700 @@ -24,7 +24,8 @@ def onPipelineStart(self, pipeline): self._map_dir = os.path.join(pipeline.tmp_dir, 'less') - if not os.path.isdir(self._map_dir): + if (pipeline.is_first_worker and + not os.path.isdir(self._map_dir)): os.makedirs(self._map_dir) def getDependencies(self, path): @@ -42,6 +43,7 @@ # Get the sources, but make all paths absolute. sources = dep_map.get('sources') path_dir = os.path.dirname(path) + def _makeAbs(p): return os.path.join(path_dir, p) deps = list(map(_makeAbs, sources))
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/piecrust/processing/pipeline.py Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,297 @@ +import os +import os.path +import re +import time +import hashlib +import logging +import multiprocessing +from piecrust.chefutil import format_timed, format_timed_scope +from piecrust.processing.base import PipelineContext +from piecrust.processing.records import ( + ProcessorPipelineRecordEntry, TransitionalProcessorPipelineRecord, + FLAG_PROCESSED) +from piecrust.processing.worker import ( + ProcessingWorkerJob, + get_filtered_processors) + + +logger = logging.getLogger(__name__) + + +class _ProcessingContext(object): + def __init__(self, jobs, record, base_dir, mount_info): + self.jobs = jobs + self.record = record + self.base_dir = base_dir + self.mount_info = mount_info + + +class ProcessorPipeline(object): + def __init__(self, app, out_dir, force=False): + assert app and out_dir + self.app = app + self.out_dir = out_dir + self.force = force + + tmp_dir = app.sub_cache_dir + if not tmp_dir: + import tempfile + tmp_dir = os.path.join(tempfile.gettempdir(), 'piecrust') + self.tmp_dir = os.path.join(tmp_dir, 'proc') + + baker_params = app.config.get('baker') or {} + + assets_dirs = baker_params.get('assets_dirs', app.assets_dirs) + self.mounts = make_mount_infos(assets_dirs, self.app.root_dir) + + self.num_workers = baker_params.get( + 'workers', multiprocessing.cpu_count()) + + ignores = baker_params.get('ignore', []) + ignores += [ + '_cache', '_counter', + 'theme_info.yml', + '.DS_Store', 'Thumbs.db', + '.git*', '.hg*', '.svn'] + self.ignore_patterns = make_re(ignores) + self.force_patterns = make_re(baker_params.get('force', [])) + + # Those things are mostly for unit-testing. + self.enabled_processors = None + self.additional_processors = None + + def addIgnorePatterns(self, patterns): + self.ignore_patterns += make_re(patterns) + + def run(self, src_dir_or_file=None, *, + delete=True, previous_record=None, save_record=True): + start_time = time.perf_counter() + + # Get the list of processors for this run. + processors = self.app.plugin_loader.getProcessors() + if self.enabled_processors is not None: + logger.debug("Filtering processors to: %s" % + self.enabled_processors) + processors = get_filtered_processors(processors, + self.enabled_processors) + if self.additional_processors is not None: + logger.debug("Adding %s additional processors." % + len(self.additional_processors)) + for proc in self.additional_processors: + self.app.env.registerTimer(proc.__class__.__name__, + raise_if_registered=False) + proc.initialize(self.app) + processors.append(proc) + + # Invoke pre-processors. + pipeline_ctx = PipelineContext(-1, self.app, self.out_dir, + self.tmp_dir, self.force) + for proc in processors: + proc.onPipelineStart(pipeline_ctx) + + # Pre-processors can define additional ignore patterns. + self.ignore_patterns += make_re( + pipeline_ctx._additional_ignore_patterns) + + # Create the pipeline record. + record = TransitionalProcessorPipelineRecord() + record_cache = self.app.cache.getCache('proc') + record_name = ( + hashlib.md5(self.out_dir.encode('utf8')).hexdigest() + + '.record') + if previous_record: + record.setPrevious(previous_record) + elif not self.force and record_cache.has(record_name): + with format_timed_scope(logger, 'loaded previous bake record', + level=logging.DEBUG, colored=False): + record.loadPrevious(record_cache.getCachePath(record_name)) + logger.debug("Got %d entries in process record." % + len(record.previous.entries)) + record.current.success = True + record.current.processed_count = 0 + + # Work! + def _handler(res): + entry = record.getCurrentEntry(res.path) + assert entry is not None + entry.flags |= res.flags + entry.proc_tree = res.proc_tree + entry.rel_outputs = res.rel_outputs + if entry.flags & FLAG_PROCESSED: + record.current.processed_count += 1 + if res.errors: + entry.errors += res.errors + record.current.success = False + + rel_path = os.path.relpath(res.path, self.app.root_dir) + logger.error("Errors found in %s:" % rel_path) + for e in entry.errors: + logger.error(" " + e) + + jobs = [] + self._process(src_dir_or_file, record, jobs) + pool = self._createWorkerPool() + ar = pool.queueJobs(jobs, handler=_handler) + ar.wait() + + # Shutdown the workers and get timing information from them. + reports = pool.close() + record.current.timers = {} + for i in range(len(reports)): + timers = reports[i] + if timers is None: + continue + + worker_name = 'PipelineWorker_%d' % i + record.current.timers[worker_name] = {} + for name, val in timers['data'].items(): + main_val = record.current.timers.setdefault(name, 0) + record.current.timers[name] = main_val + val + record.current.timers[worker_name][name] = val + + # Invoke post-processors. + pipeline_ctx.record = record.current + for proc in processors: + proc.onPipelineEnd(pipeline_ctx) + + # Handle deletions. + if delete: + for path, reason in record.getDeletions(): + logger.debug("Removing '%s': %s" % (path, reason)) + try: + os.remove(path) + except FileNotFoundError: + pass + logger.info('[delete] %s' % path) + + # Finalize the process record. + record.current.process_time = time.time() + record.current.out_dir = self.out_dir + record.collapseRecords() + + # Save the process record. + if save_record: + with format_timed_scope(logger, 'saved bake record', + level=logging.DEBUG, colored=False): + record.saveCurrent(record_cache.getCachePath(record_name)) + + logger.info(format_timed( + start_time, + "processed %d assets." % record.current.processed_count)) + + return record.detach() + + def _process(self, src_dir_or_file, record, jobs): + if src_dir_or_file is not None: + # Process only the given path. + # Find out what mount point this is in. + for name, info in self.mounts.items(): + path = info['path'] + if src_dir_or_file[:len(path)] == path: + base_dir = path + mount_info = info + break + else: + known_roots = [i['path'] for i in self.mounts.values()] + raise Exception("Input path '%s' is not part of any known " + "mount point: %s" % + (src_dir_or_file, known_roots)) + + ctx = _ProcessingContext(jobs, record, base_dir, mount_info) + logger.debug("Initiating processing pipeline on: %s" % + src_dir_or_file) + if os.path.isdir(src_dir_or_file): + self._processDirectory(ctx, src_dir_or_file) + elif os.path.isfile(src_dir_or_file): + self._processFile(ctx, src_dir_or_file) + + else: + # Process everything. + for name, info in self.mounts.items(): + path = info['path'] + ctx = _ProcessingContext(jobs, record, path, info) + logger.debug("Initiating processing pipeline on: %s" % path) + self._processDirectory(ctx, path) + + def _processDirectory(self, ctx, start_dir): + for dirpath, dirnames, filenames in os.walk(start_dir): + rel_dirpath = os.path.relpath(dirpath, start_dir) + dirnames[:] = [d for d in dirnames + if not re_matchany( + d, self.ignore_patterns, rel_dirpath)] + + for filename in filenames: + if re_matchany(filename, self.ignore_patterns, rel_dirpath): + continue + self._processFile(ctx, os.path.join(dirpath, filename)) + + def _processFile(self, ctx, path): + # TODO: handle overrides between mount-points. + + entry = ProcessorPipelineRecordEntry(path) + ctx.record.addEntry(entry) + + previous_entry = ctx.record.getPreviousEntry(path) + force_this = (self.force or previous_entry is None or + not previous_entry.was_processed_successfully) + + job = ProcessingWorkerJob(ctx.base_dir, ctx.mount_info, path, + force=force_this) + ctx.jobs.append(job) + + def _createWorkerPool(self): + from piecrust.workerpool import WorkerPool + from piecrust.processing.worker import ( + ProcessingWorkerContext, ProcessingWorker) + + ctx = ProcessingWorkerContext( + self.app.root_dir, self.out_dir, self.tmp_dir, + self.force, self.app.debug) + ctx.enabled_processors = self.enabled_processors + ctx.additional_processors = self.additional_processors + + pool = WorkerPool( + worker_class=ProcessingWorker, + initargs=(ctx,)) + return pool + + +def make_mount_infos(mounts, root_dir): + if isinstance(mounts, list): + mounts = {m: {} for m in mounts} + + for name, info in mounts.items(): + if not isinstance(info, dict): + raise Exception("Asset directory info for '%s' is not a " + "dictionary." % name) + info.setdefault('processors', 'all -uglifyjs -cleancss') + info['path'] = os.path.join(root_dir, name) + + return mounts + + +def make_re(patterns): + re_patterns = [] + for pat in patterns: + if pat[0] == '/' and pat[-1] == '/' and len(pat) > 2: + re_patterns.append(pat[1:-1]) + else: + escaped_pat = ( + re.escape(pat) + .replace(r'\*', r'[^/\\]*') + .replace(r'\?', r'[^/\\]')) + re_patterns.append(escaped_pat) + return [re.compile(p) for p in re_patterns] + + +def re_matchany(filename, patterns, dirname=None): + if dirname and dirname != '.': + filename = os.path.join(dirname, filename) + + # skip patterns use a forward slash regardless of the platform. + filename = filename.replace('\\', '/') + for pattern in patterns: + if pattern.search(filename): + return True + return False +
--- a/piecrust/processing/records.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/processing/records.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,48 +1,33 @@ import os.path +import hashlib from piecrust.records import Record, TransitionalRecord class ProcessorPipelineRecord(Record): - RECORD_VERSION = 4 + RECORD_VERSION = 5 def __init__(self): super(ProcessorPipelineRecord, self).__init__() self.out_dir = None self.process_time = None + self.processed_count = 0 self.success = False - - def hasOverrideEntry(self, rel_path): - return self.findEntry(rel_path) is not None - - def findEntry(self, rel_path): - rel_path = rel_path.lower() - for entry in self.entries: - for out_path in entry.rel_outputs: - if out_path.lower() == rel_path: - return entry - return None - - def replaceEntry(self, new_entry): - for e in self.entries: - if (e.base_dir == new_entry.base_dir and - e.rel_input == new_entry.rel_input): - e.flags = new_entry.flags - e.rel_outputs = list(new_entry.rel_outputs) - e.errors = list(new_entry.errors) - break + self.timers = None FLAG_NONE = 0 FLAG_PREPARED = 2**0 FLAG_PROCESSED = 2**1 -FLAG_OVERRIDEN = 2**2 FLAG_BYPASSED_STRUCTURED_PROCESSING = 2**3 +def _get_transition_key(path): + return hashlib.md5(path.encode('utf8')).hexdigest() + + class ProcessorPipelineRecordEntry(object): - def __init__(self, base_dir, rel_input): - self.base_dir = base_dir - self.rel_input = rel_input + def __init__(self, path): + self.path = path self.flags = FLAG_NONE self.rel_outputs = [] @@ -50,10 +35,6 @@ self.errors = [] @property - def path(self): - return os.path.join(self.base_dir, self.rel_input) - - @property def was_prepared(self): return bool(self.flags & FLAG_PREPARED) @@ -73,10 +54,18 @@ ProcessorPipelineRecord, previous_path) def getTransitionKey(self, entry): - return entry.rel_input + return _get_transition_key(entry.path) - def getPreviousEntry(self, rel_path): - pair = self.transitions.get(rel_path) + def getCurrentEntry(self, path): + key = _get_transition_key(path) + pair = self.transitions.get(key) + if pair is not None: + return pair[1] + return None + + def getPreviousEntry(self, path): + key = _get_transition_key(path) + pair = self.transitions.get(key) if pair is not None: return pair[0] return None
--- a/piecrust/processing/sass.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/processing/sass.py Sat Jul 11 20:33:55 2015 -0700 @@ -26,12 +26,14 @@ def onPipelineStart(self, pipeline): super(SassProcessor, self).onPipelineStart(pipeline) - self._map_dir = os.path.join(pipeline.tmp_dir, 'sass') - if not os.path.isdir(self._map_dir): - os.makedirs(self._map_dir) + + if pipeline.is_first_worker: + self._map_dir = os.path.join(pipeline.tmp_dir, 'sass') + if not os.path.isdir(self._map_dir): + os.makedirs(self._map_dir) # Ignore include-only Sass files. - pipeline.addSkipPatterns(['_*.scss', '_*.sass']) + pipeline.addIgnorePatterns(['_*.scss', '_*.sass']) def getDependencies(self, path): if _is_include_only(path):
--- a/piecrust/processing/sitemap.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/processing/sitemap.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,7 +1,9 @@ import time import logging import yaml +from piecrust.data.iterators import PageIterator from piecrust.processing.base import SimpleFileProcessor +from piecrust.routing import create_route_metadata logger = logging.getLogger(__name__) @@ -9,8 +11,7 @@ SITEMAP_HEADER = \ """<?xml version="1.0" encoding="utf-8"?> -<urlset - xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> +<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> """ SITEMAP_FOOTER = "</urlset>\n" @@ -18,7 +19,7 @@ SITEURL_LOC = " <loc>%s</loc>\n" SITEURL_LASTMOD = " <lastmod>%s</lastmod>\n" SITEURL_CHANGEFREQ = " <changefreq>%s</changefreq>\n" -SITEURL_PRIORITY = " <priority>%f</priority>\n" +SITEURL_PRIORITY = " <priority>%0.1f</priority>\n" SITEURL_FOOTER = " </url>\n" @@ -58,21 +59,20 @@ if not source_names: return + cur_time = strftime_iso8601(time.time()) for name in source_names: logger.debug("Generating automatic sitemap entries for '%s'." % - name) + name) source = self.app.getSource(name) if source is None: raise Exception("No such source: %s" % name) - for page in source.getPages(): - route = self.app.getRoute(source.name, page.source_metadata) - uri = route.getUri(page.source_metadata, provider=page) + it = PageIterator(source) + for page in it: + uri = page['url'] + sm_cfg = page.get('sitemap') - t = page.datetime.timestamp() - sm_cfg = page.config.get('sitemap') - - args = {'url': uri, 'lastmod': strftime_iso8601(t)} + args = {'url': uri, 'lastmod': cur_time} if sm_cfg: args.update(sm_cfg)
--- a/piecrust/processing/tree.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/processing/tree.py Sat Jul 11 20:33:55 2015 -0700 @@ -79,7 +79,6 @@ self.processors = processors def build(self, path): - start_time = time.clock() tree_root = ProcessingTreeNode(path, list(self.processors)) loop_guard = 100 @@ -97,7 +96,7 @@ if proc.is_bypassing_structured_processing: if cur_node != tree_root: raise ProcessingTreeError("Only root processors can " - "bypass structured processing.") + "bypass structured processing.") break # Get the destination directory and output files. @@ -116,18 +115,14 @@ if proc.PROCESSOR_NAME != 'copy': walk_stack.append(out_node) - logger.debug(format_timed( - start_time, "Built processing tree for: %s" % path, - colored=False)) return tree_root class ProcessingTreeRunner(object): - def __init__(self, base_dir, tmp_dir, out_dir, lock=None): + def __init__(self, base_dir, tmp_dir, out_dir): self.base_dir = base_dir self.tmp_dir = tmp_dir self.out_dir = out_dir - self.lock = lock def processSubTree(self, tree_root): did_process = False @@ -155,8 +150,9 @@ proc = node.getProcessor() if proc.is_bypassing_structured_processing: try: - start_time = time.clock() - proc.process(full_path, self.out_dir) + start_time = time.perf_counter() + with proc.app.env.timerScope(proc.__class__.__name__): + proc.process(full_path, self.out_dir) print_node( node, format_timed( @@ -172,16 +168,15 @@ rel_out_dir = os.path.dirname(node.path) out_dir = os.path.join(base_out_dir, rel_out_dir) if not os.path.isdir(out_dir): - if self.lock: - with self.lock: - if not os.path.isdir(out_dir): - os.makedirs(out_dir, 0o755) - else: - os.makedirs(out_dir, 0o755) + try: + os.makedirs(out_dir, 0o755, exist_ok=True) + except OSError: + pass try: - start_time = time.clock() - proc_res = proc.process(full_path, out_dir) + start_time = time.perf_counter() + with proc.app.env.timerScope(proc.__class__.__name__): + proc_res = proc.process(full_path, out_dir) if proc_res is None: raise Exception("Processor '%s' didn't return a boolean " "result value." % proc) @@ -200,12 +195,12 @@ proc = node.getProcessor() if (proc.is_bypassing_structured_processing or - not proc.is_delegating_dependency_check): + not proc.is_delegating_dependency_check): # This processor wants to handle things on its own... node.setState(STATE_DIRTY, False) return - start_time = time.clock() + start_time = time.perf_counter() # Get paths and modification times for the input path and # all dependencies (if any).
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/piecrust/processing/worker.py Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,184 @@ +import re +import os.path +import time +import logging +from piecrust.app import PieCrust +from piecrust.processing.base import PipelineContext +from piecrust.processing.records import ( + FLAG_NONE, FLAG_PREPARED, FLAG_PROCESSED, + FLAG_BYPASSED_STRUCTURED_PROCESSING) +from piecrust.processing.tree import ( + ProcessingTreeBuilder, ProcessingTreeRunner, + ProcessingTreeError, ProcessorError, + get_node_name_tree, print_node, + STATE_DIRTY) +from piecrust.workerpool import IWorker + + +logger = logging.getLogger(__name__) + + +split_processor_names_re = re.compile(r'[ ,]+') +re_ansicolors = re.compile('\033\\[\d+m') + + +class ProcessingWorkerContext(object): + def __init__(self, root_dir, out_dir, tmp_dir, + force=False, debug=False): + self.root_dir = root_dir + self.out_dir = out_dir + self.tmp_dir = tmp_dir + self.force = force + self.debug = debug + self.is_profiling = False + self.enabled_processors = None + self.additional_processors = None + + +class ProcessingWorkerJob(object): + def __init__(self, base_dir, mount_info, path, *, force=False): + self.base_dir = base_dir + self.mount_info = mount_info + self.path = path + self.force = force + + +class ProcessingWorkerResult(object): + def __init__(self, path): + self.path = path + self.flags = FLAG_NONE + self.proc_tree = None + self.rel_outputs = None + self.errors = None + + +class ProcessingWorker(IWorker): + def __init__(self, ctx): + self.ctx = ctx + self.work_start_time = time.perf_counter() + + def initialize(self): + # Create the app local to this worker. + app = PieCrust(self.ctx.root_dir, debug=self.ctx.debug) + app.env.registerTimer("PipelineWorker_%d_Total" % self.wid) + app.env.registerTimer("PipelineWorkerInit") + app.env.registerTimer("JobReceive") + app.env.registerTimer('BuildProcessingTree') + app.env.registerTimer('RunProcessingTree') + self.app = app + + processors = app.plugin_loader.getProcessors() + if self.ctx.enabled_processors: + logger.debug("Filtering processors to: %s" % + self.ctx.enabled_processors) + processors = get_filtered_processors(processors, + self.ctx.enabled_processors) + if self.ctx.additional_processors: + logger.debug("Adding %s additional processors." % + len(self.ctx.additional_processors)) + for proc in self.ctx.additional_processors: + app.env.registerTimer(proc.__class__.__name__) + proc.initialize(app) + processors.append(proc) + self.processors = processors + + # Invoke pre-processors. + pipeline_ctx = PipelineContext(self.wid, self.app, self.ctx.out_dir, + self.ctx.tmp_dir, self.ctx.force) + for proc in processors: + proc.onPipelineStart(pipeline_ctx) + + # Sort our processors again in case the pre-process step involved + # patching the processors with some new ones. + processors.sort(key=lambda p: p.priority) + + app.env.stepTimerSince("PipelineWorkerInit", self.work_start_time) + + def process(self, job): + result = ProcessingWorkerResult(job.path) + + processors = get_filtered_processors( + self.processors, job.mount_info['processors']) + + # Build the processing tree for this job. + rel_path = os.path.relpath(job.path, job.base_dir) + try: + with self.app.env.timerScope('BuildProcessingTree'): + builder = ProcessingTreeBuilder(processors) + tree_root = builder.build(rel_path) + result.flags |= FLAG_PREPARED + except ProcessingTreeError as ex: + result.errors = _get_errors(ex) + return result + + # Prepare and run the tree. + print_node(tree_root, recursive=True) + leaves = tree_root.getLeaves() + result.rel_outputs = [l.path for l in leaves] + result.proc_tree = get_node_name_tree(tree_root) + if tree_root.getProcessor().is_bypassing_structured_processing: + result.flags |= FLAG_BYPASSED_STRUCTURED_PROCESSING + + if job.force: + tree_root.setState(STATE_DIRTY, True) + + try: + with self.app.env.timerScope('RunProcessingTree'): + runner = ProcessingTreeRunner( + job.base_dir, self.ctx.tmp_dir, self.ctx.out_dir) + if runner.processSubTree(tree_root): + result.flags |= FLAG_PROCESSED + except ProcessingTreeError as ex: + if isinstance(ex, ProcessorError): + ex = ex.__cause__ + # Need to strip out colored errors from external processes. + result.errors = _get_errors(ex, strip_colors=True) + + return result + + def getReport(self): + # Invoke post-processors. + pipeline_ctx = PipelineContext(self.wid, self.app, self.ctx.out_dir, + self.ctx.tmp_dir, self.ctx.force) + for proc in self.processors: + proc.onPipelineEnd(pipeline_ctx) + + self.app.env.stepTimerSince("PipelineWorker_%d_Total" % self.wid, + self.work_start_time) + return { + 'type': 'timers', + 'data': self.app.env._timers} + + +def get_filtered_processors(processors, authorized_names): + if not authorized_names or authorized_names == 'all': + return processors + + if isinstance(authorized_names, str): + authorized_names = split_processor_names_re.split(authorized_names) + + procs = [] + has_star = 'all' in authorized_names + for p in processors: + for name in authorized_names: + if name == p.PROCESSOR_NAME: + procs.append(p) + break + if name == ('-%s' % p.PROCESSOR_NAME): + break + else: + if has_star: + procs.append(p) + return procs + + +def _get_errors(ex, strip_colors=False): + errors = [] + while ex is not None: + msg = str(ex) + if strip_colors: + msg = re_ansicolors.sub('', msg) + errors.append(msg) + ex = ex.__cause__ + return errors +
--- a/piecrust/rendering.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/rendering.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,9 +1,10 @@ import re import os.path +import copy import logging from werkzeug.utils import cached_property -from piecrust.data.builder import (DataBuildingContext, build_page_data, - build_layout_data) +from piecrust.data.builder import ( + DataBuildingContext, build_page_data, build_layout_data) from piecrust.data.filters import ( PaginationFilter, HasFilterClause, IsFilterClause, AndBooleanClause, page_value_accessor) @@ -33,13 +34,24 @@ self.route_metadata = route_metadata def getUri(self, sub_num=1): - return self.route.getUri(self.route_metadata, provider=self.page, - sub_num=sub_num) + return self.route.getUri(self.route_metadata, sub_num=sub_num) def __getattr__(self, name): return getattr(self.page, name) +class RenderedSegments(object): + def __init__(self, segments, render_pass_info): + self.segments = segments + self.render_pass_info = render_pass_info + + +class RenderedLayout(object): + def __init__(self, content, render_pass_info): + self.content = content + self.render_pass_info = render_pass_info + + class RenderedPage(object): def __init__(self, page, uri, num=1): self.page = page @@ -47,11 +59,15 @@ self.num = num self.data = None self.content = None + self.render_info = None @property def app(self): return self.page.app + def copyRenderInfo(self): + return copy.deepcopy(self.render_info) + PASS_NONE = 0 PASS_FORMATTING = 1 @@ -65,6 +81,41 @@ def __init__(self): self.used_source_names = set() self.used_taxonomy_terms = set() + self.used_pagination = False + self.pagination_has_more = False + self.used_assets = False + + def merge(self, other): + self.used_source_names |= other.used_source_names + self.used_taxonomy_terms |= other.used_taxonomy_terms + self.used_pagination = self.used_pagination or other.used_pagination + self.pagination_has_more = (self.pagination_has_more or + other.pagination_has_more) + self.used_assets = self.used_assets or other.used_assets + + def _toJson(self): + data = { + 'used_source_names': list(self.used_source_names), + 'used_taxonomy_terms': list(self.used_taxonomy_terms), + 'used_pagination': self.used_pagination, + 'pagination_has_more': self.pagination_has_more, + 'used_assets': self.used_assets} + return data + + @staticmethod + def _fromJson(data): + assert data is not None + rpi = RenderPassInfo() + rpi.used_source_names = set(data['used_source_names']) + for i in data['used_taxonomy_terms']: + terms = i[2] + if isinstance(terms, list): + terms = tuple(terms) + rpi.used_taxonomy_terms.add((i[0], i[1], terms)) + rpi.used_pagination = data['used_pagination'] + rpi.pagination_has_more = data['pagination_has_more'] + rpi.used_assets = data['used_assets'] + return rpi class PageRenderingContext(object): @@ -78,8 +129,6 @@ self._current_pass = PASS_NONE self.render_passes = {} - self.used_pagination = None - self.used_assets = None @property def app(self): @@ -104,15 +153,18 @@ def setPagination(self, paginator): self._raiseIfNoCurrentPass() - if self.used_pagination is not None: + pass_info = self.current_pass_info + if pass_info.used_pagination: raise Exception("Pagination has already been used.") - self.used_pagination = paginator + assert paginator.is_loaded + pass_info.used_pagination = True + pass_info.pagination_has_more = paginator.has_more self.addUsedSource(paginator._source) def addUsedSource(self, source): self._raiseIfNoCurrentPass() if isinstance(source, PageSource): - pass_info = self.render_passes[self._current_pass] + pass_info = self.current_pass_info pass_info.used_source_names.add(source.name) def setTaxonomyFilter(self, taxonomy, term_value): @@ -144,44 +196,50 @@ eis = ctx.app.env.exec_info_stack eis.pushPage(ctx.page, ctx) try: - page = ctx.page - # Build the data for both segment and layout rendering. - data_ctx = DataBuildingContext(page, page_num=ctx.page_num) - data_ctx.pagination_source = ctx.pagination_source - data_ctx.pagination_filter = ctx.pagination_filter - page_data = build_page_data(data_ctx) - if ctx.custom_data: - page_data.update(ctx.custom_data) + page_data = _build_render_data(ctx) # Render content segments. ctx.setCurrentPass(PASS_FORMATTING) repo = ctx.app.env.rendered_segments_repository + save_to_fs = True + if ctx.app.env.fs_cache_only_for_main_page and not eis.is_main_page: + save_to_fs = False if repo and not ctx.force_render: - cache_key = ctx.uri - page_time = page.path_mtime - contents = repo.get( - cache_key, - lambda: _do_render_page_segments(page, page_data), - fs_cache_time=page_time) + render_result = repo.get( + ctx.uri, + lambda: _do_render_page_segments(ctx.page, page_data), + fs_cache_time=ctx.page.path_mtime, + save_to_fs=save_to_fs) else: - contents = _do_render_page_segments(page, page_data) + render_result = _do_render_page_segments(ctx.page, page_data) + if repo: + repo.put(ctx.uri, render_result, save_to_fs) # Render layout. + page = ctx.page ctx.setCurrentPass(PASS_RENDERING) layout_name = page.config.get('layout') if layout_name is None: layout_name = page.source.config.get('default_layout', 'default') null_names = ['', 'none', 'nil'] if layout_name not in null_names: - build_layout_data(page, page_data, contents) - output = render_layout(layout_name, page, page_data) + build_layout_data(page, page_data, render_result['segments']) + layout_result = _do_render_layout(layout_name, page, page_data) else: - output = contents['content'] + layout_result = { + 'content': render_result['segments']['content'], + 'pass_info': None} rp = RenderedPage(page, ctx.uri, ctx.page_num) rp.data = page_data - rp.content = output + rp.content = layout_result['content'] + rp.render_info = { + PASS_FORMATTING: RenderPassInfo._fromJson( + render_result['pass_info'])} + if layout_result['pass_info'] is not None: + rp.render_info[PASS_RENDERING] = RenderPassInfo._fromJson( + layout_result['pass_info']) return rp finally: ctx.setCurrentPass(PASS_NONE) @@ -189,69 +247,100 @@ def render_page_segments(ctx): - repo = ctx.app.env.rendered_segments_repository - if repo: - cache_key = ctx.uri - return repo.get( - cache_key, - lambda: _do_render_page_segments_from_ctx(ctx), - fs_cache_time=ctx.page.path_mtime) - - return _do_render_page_segments_from_ctx(ctx) - - -def _do_render_page_segments_from_ctx(ctx): eis = ctx.app.env.exec_info_stack eis.pushPage(ctx.page, ctx) - ctx.setCurrentPass(PASS_FORMATTING) try: - data_ctx = DataBuildingContext(ctx.page, page_num=ctx.page_num) - page_data = build_page_data(data_ctx) - return _do_render_page_segments(ctx.page, page_data) + ctx.setCurrentPass(PASS_FORMATTING) + repo = ctx.app.env.rendered_segments_repository + save_to_fs = True + if ctx.app.env.fs_cache_only_for_main_page and not eis.is_main_page: + save_to_fs = False + if repo and not ctx.force_render: + render_result = repo.get( + ctx.uri, + lambda: _do_render_page_segments_from_ctx(ctx), + fs_cache_time=ctx.page.path_mtime, + save_to_fs=save_to_fs) + else: + render_result = _do_render_page_segments_from_ctx(ctx) + if repo: + repo.put(ctx.uri, render_result, save_to_fs) finally: ctx.setCurrentPass(PASS_NONE) eis.popPage() + rs = RenderedSegments( + render_result['segments'], + RenderPassInfo._fromJson(render_result['pass_info'])) + return rs + + +def _build_render_data(ctx): + with ctx.app.env.timerScope("PageDataBuild"): + data_ctx = DataBuildingContext(ctx.page, page_num=ctx.page_num) + data_ctx.pagination_source = ctx.pagination_source + data_ctx.pagination_filter = ctx.pagination_filter + page_data = build_page_data(data_ctx) + if ctx.custom_data: + page_data._appendMapping(ctx.custom_data) + return page_data + + +def _do_render_page_segments_from_ctx(ctx): + page_data = _build_render_data(ctx) + return _do_render_page_segments(ctx.page, page_data) + def _do_render_page_segments(page, page_data): app = page.app + + cpi = app.env.exec_info_stack.current_page_info + assert cpi is not None + assert cpi.page == page + engine_name = page.config.get('template_engine') format_name = page.config.get('format') engine = get_template_engine(app, engine_name) - formatted_content = {} + formatted_segments = {} for seg_name, seg in page.raw_content.items(): seg_text = '' for seg_part in seg.parts: part_format = seg_part.fmt or format_name try: - part_text = engine.renderString( - seg_part.content, page_data, - filename=page.path) + with app.env.timerScope(engine.__class__.__name__): + part_text = engine.renderSegmentPart( + page.path, seg_part, page_data) except TemplatingError as err: err.lineno += seg_part.line raise err part_text = format_text(app, part_format, part_text) seg_text += part_text - formatted_content[seg_name] = seg_text + formatted_segments[seg_name] = seg_text if seg_name == 'content': m = content_abstract_re.search(seg_text) if m: offset = m.start() content_abstract = seg_text[:offset] - formatted_content['content.abstract'] = content_abstract + formatted_segments['content.abstract'] = content_abstract - return formatted_content + pass_info = cpi.render_ctx.render_passes.get(PASS_FORMATTING) + res = { + 'segments': formatted_segments, + 'pass_info': pass_info._toJson()} + return res -def render_layout(layout_name, page, layout_data): +def _do_render_layout(layout_name, page, layout_data): + cpi = page.app.env.exec_info_stack.current_page_info + assert cpi is not None + assert cpi.page == page + names = layout_name.split(',') - default_template_engine = get_template_engine(page.app, None) - default_exts = ['.' + e.lstrip('.') - for e in default_template_engine.EXTENSIONS] + default_exts = page.app.env.default_layout_extensions full_names = [] for name in names: if '.' not in name: @@ -270,7 +359,10 @@ msg = "Can't find template for page: %s\n" % page.path msg += "Looked for: %s" % ', '.join(full_names) raise Exception(msg) from ex - return output + + pass_info = cpi.render_ctx.render_passes.get(PASS_RENDERING) + res = {'content': output, 'pass_info': pass_info._toJson()} + return res def get_template_engine(app, engine_name): @@ -290,8 +382,11 @@ format_count = 0 format_name = format_name or app.config.get('site/default_format') for fmt in app.plugin_loader.getFormatters(): + if not fmt.enabled: + continue if fmt.FORMAT_NAMES is None or format_name in fmt.FORMAT_NAMES: - txt = fmt.render(format_name, txt) + with app.env.timerScope(fmt.__class__.__name__): + txt = fmt.render(format_name, txt) format_count += 1 if fmt.OUTPUT_FORMAT is not None: format_name = fmt.OUTPUT_FORMAT
--- a/piecrust/routing.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/routing.py Sat Jul 11 20:33:55 2015 -0700 @@ -15,6 +15,18 @@ ugly_url_cleaner = re.compile(r'\.html$') +def create_route_metadata(page): + route_metadata = copy.deepcopy(page.source_metadata) + route_metadata.update(page.getRouteMetadata()) + + # TODO: fix this hard-coded shit + for key in ['year', 'month', 'day']: + if key in route_metadata and isinstance(route_metadata[key], str): + route_metadata[key] = int(route_metadata[key]) + + return route_metadata + + class IRouteMetadataProvider(object): def getRouteMetadata(self): raise NotImplementedError() @@ -120,18 +132,14 @@ for k in missing_keys: route_metadata[k] = '' - return route_metadata - - def getUri(self, route_metadata, *, sub_num=1, provider=None): - route_metadata = copy.deepcopy(route_metadata) - if provider: - route_metadata.update(provider.getRouteMetadata()) - - #TODO: fix this hard-coded shit + # TODO: fix this hard-coded shit for key in ['year', 'month', 'day']: if key in route_metadata and isinstance(route_metadata[key], str): route_metadata[key] = int(route_metadata[key]) + return route_metadata + + def getUri(self, route_metadata, *, sub_num=1): uri = self.uri_format % route_metadata suffix = None if sub_num > 1: @@ -280,6 +288,9 @@ len(args))) metadata = {} for arg_name, arg_val in zip(self.template_func_args, args): + #TODO: fix this hard-coded shit. + if arg_name in ['year', 'month', 'day']: + arg_val = int(arg_val) metadata[arg_name] = arg_val return self.getUri(metadata)
--- a/piecrust/serving/server.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/serving/server.py Sat Jul 11 20:33:55 2015 -0700 @@ -12,7 +12,7 @@ from werkzeug.wrappers import Request, Response from werkzeug.wsgi import ClosingIterator, wrap_file from jinja2 import FileSystemLoader, Environment -from piecrust import CACHE_DIR +from piecrust import CACHE_DIR, RESOURCES_DIR from piecrust.app import PieCrust from piecrust.rendering import QualifiedPage, PageRenderingContext, render_page from piecrust.sources.base import MODE_PARSING @@ -81,7 +81,7 @@ # use process forking and we end up going here twice. We only want # to start the pipeline loop in the inner process most of the # time so we let the implementation tell us if this is OK. - from piecrust.processing.base import ProcessorPipeline + from piecrust.processing.pipeline import ProcessorPipeline from piecrust.serving.procloop import ProcessingLoop pipeline = ProcessorPipeline(app, self._out_dir) self._proc_loop = ProcessingLoop(pipeline) @@ -96,6 +96,8 @@ return self._try_run_request(environ, start_response) except Exception as ex: if self.debug: + if isinstance(ex, HTTPException): + return ex raise return self._handle_error(ex, environ, start_response) @@ -158,9 +160,7 @@ static_mount = '/__piecrust_static/' if request.path.startswith(static_mount): rel_req_path = request.path[len(static_mount):] - mount = os.path.join( - os.path.dirname(__file__), - 'resources', 'server') + mount = os.path.join(RESOURCES_DIR, 'server') full_path = os.path.join(mount, rel_req_path) try: response = self._make_wrapped_file_response( @@ -173,7 +173,7 @@ if request.path.startswith(debug_mount): rel_req_path = request.path[len(debug_mount):] if rel_req_path == 'pipeline_status': - from piecrust.server.procloop import ( + from piecrust.serving.procloop import ( PipelineStatusServerSideEventProducer) provider = PipelineStatusServerSideEventProducer( self._proc_loop.status_queue) @@ -321,7 +321,7 @@ if route_terms is None: return None - tax_page_ref = taxonomy.getPageRef(source.name) + tax_page_ref = taxonomy.getPageRef(source) factory = tax_page_ref.getFactory() tax_terms = route.unslugifyTaxonomyTerm(route_terms) route_metadata[taxonomy.term_name] = tax_terms @@ -459,8 +459,7 @@ class ErrorMessageLoader(FileSystemLoader): def __init__(self): - base_dir = os.path.join(os.path.dirname(__file__), 'resources', - 'messages') + base_dir = os.path.join(RESOURCES_DIR, 'messages') super(ErrorMessageLoader, self).__init__(base_dir) def get_source(self, env, template):
--- a/piecrust/sources/array.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/sources/array.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,5 +1,6 @@ from piecrust.sources.base import PageSource from piecrust.sources.mixins import SimplePaginationSourceMixin +from piecrust.sources.pageref import PageRef class CachedPageFactory(object): @@ -30,7 +31,7 @@ class ArraySource(PageSource, SimplePaginationSourceMixin): def __init__(self, app, inner_source, name='array', config=None): - super(ArraySource, self).__init__(app, name, config or {}) + super(ArraySource, self).__init__(app, name, config) self.inner_source = inner_source @property @@ -41,3 +42,6 @@ for p in self.inner_source: yield CachedPageFactory(p) + def getTaxonomyPageRef(self, tax_name): + return None +
--- a/piecrust/sources/base.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/sources/base.py Sat Jul 11 20:33:55 2015 -0700 @@ -20,9 +20,8 @@ def build_pages(app, factories): - with app.env.page_repository.startBatchGet(): - for f in factories: - yield f.buildPage() + for f in factories: + yield f.buildPage() class InvalidFileSystemEndpointError(Exception): @@ -59,9 +58,6 @@ def _doBuildPage(self): logger.debug("Building page: %s" % self.path) page = Page(self.source, copy.deepcopy(self.metadata), self.rel_path) - # Load it right away, especially when using the page repository, - # because we'll be inside a critical scope. - page._load() return page @@ -118,7 +114,7 @@ def findPageFactory(self, metadata, mode): raise NotImplementedError() - def buildDataProvider(self, page, user_data): + def buildDataProvider(self, page, override): if self._provider_type is None: cls = next((pt for pt in self.app.plugin_loader.getDataProviders() if pt.PROVIDER_NAME == self.data_type), @@ -128,7 +124,7 @@ "Unknown data provider type: %s" % self.data_type) self._provider_type = cls - return self._provider_type(self, page, user_data) + return self._provider_type(self, page, override) def getTaxonomyPageRef(self, tax_name): tax_pages = self.config.get('taxonomy_pages')
--- a/piecrust/sources/mixins.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/sources/mixins.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,11 +1,11 @@ import os import os.path import logging -from piecrust.data.base import PaginationData from piecrust.data.filters import PaginationFilter, page_value_accessor +from piecrust.data.paginationdata import PaginationData from piecrust.sources.base import PageFactory from piecrust.sources.interfaces import IPaginationSource, IListableSource -from piecrust.sources.pageref import PageNotFoundError +from piecrust.sources.pageref import PageRef logger = logging.getLogger(__name__) @@ -41,13 +41,15 @@ if self._taxonomy_pages is not None: return + app = self.source.app self._taxonomy_pages = set() - for tax in self.source.app.taxonomies: - page_ref = tax.getPageRef(self.source.name) - try: - self._taxonomy_pages.add(page_ref.rel_path) - except PageNotFoundError: - pass + for src in app.sources: + for tax in app.taxonomies: + ref_spec = src.getTaxonomyPageRef(tax.name) + page_ref = PageRef(app, ref_spec) + for sn, rp in page_ref.possible_split_ref_specs: + if sn == self.source.name: + self._taxonomy_pages.add(rp) class DateSortIterator(object):
--- a/piecrust/sources/pageref.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/sources/pageref.py Sat Jul 11 20:33:55 2015 -0700 @@ -42,7 +42,6 @@ @property def source_name(self): - self._checkHits() return self._first_valid_hit.source_name @property @@ -51,23 +50,25 @@ @property def rel_path(self): - self._checkHits() return self._first_valid_hit.rel_path @property def path(self): - self._checkHits() return self._first_valid_hit.path @property def metadata(self): - self._checkHits() return self._first_valid_hit.metadata @property - def possible_rel_paths(self): + def possible_ref_specs(self): self._load() - return [h.rel_path for h in self._hits] + return ['%s:%s' % (h.source_name, h.rel_path) for h in self._hits] + + @property + def possible_split_ref_specs(self): + self._load() + return [(h.source_name, h.rel_path) for h in self._hits] @property def possible_paths(self): @@ -80,17 +81,23 @@ @property def _first_valid_hit(self): + self._checkHits() return self._hits[self._first_valid_hit_index] def _load(self): if self._hits is not None: return + self._hits = [] + + if self._page_ref is None: + self._first_valid_hit_index = self._INDEX_NOT_FOUND + return + it = list(page_ref_pattern.finditer(self._page_ref)) if len(it) == 0: raise Exception("Invalid page ref: %s" % self._page_ref) - self._hits = [] for m in it: source_name = m.group('src') source = self.app.getSource(source_name) @@ -111,15 +118,17 @@ def _checkHits(self): if self._first_valid_hit_index >= 0: return + + if self._first_valid_hit_index == self._INDEX_NEEDS_LOADING: + self._load() + self._first_valid_hit_index = self._INDEX_NOT_FOUND + for i, hit in enumerate(self._hits): + if os.path.isfile(hit.path): + self._first_valid_hit_index = i + break + if self._first_valid_hit_index == self._INDEX_NOT_FOUND: raise PageNotFoundError( "No valid paths were found for page reference: %s" % self._page_ref) - self._load() - self._first_valid_hit_index = self._INDEX_NOT_FOUND - for i, hit in enumerate(self._hits): - if os.path.isfile(hit.path): - self._first_valid_hit_index = i - break -
--- a/piecrust/taxonomies.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/taxonomies.py Sat Jul 11 20:33:55 2015 -0700 @@ -16,21 +16,19 @@ return self.name return self.term_name - def resolvePagePath(self, source_name): - pr = self.getPageRef(source_name) + def resolvePagePath(self, source): + pr = self.getPageRef(source) try: return pr.path except PageNotFoundError: return None - def getPageRef(self, source_name): - if source_name in self._source_page_refs: - return self._source_page_refs[source_name] + def getPageRef(self, source): + if source.name in self._source_page_refs: + return self._source_page_refs[source.name] - source = self.app.getSource(source_name) - ref_path = (source.getTaxonomyPageRef(self.name) or - '%s:%s' % (source_name, self.page_ref)) + ref_path = source.getTaxonomyPageRef(self.name) page_ref = PageRef(self.app, ref_path) - self._source_page_refs[source_name] = page_ref + self._source_page_refs[source.name] = page_ref return page_ref
--- a/piecrust/templating/base.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/templating/base.py Sat Jul 11 20:33:55 2015 -0700 @@ -27,7 +27,7 @@ def initialize(self, app): self.app = app - def renderString(self, txt, data, filename=None, line_offset=0): + def renderSegmentPart(self, path, seg_part, data): raise NotImplementedError() def renderFile(self, paths, data):
--- a/piecrust/templating/jinjaengine.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/templating/jinjaengine.py Sat Jul 11 20:33:55 2015 -0700 @@ -33,13 +33,19 @@ def __init__(self): self.env = None - def renderString(self, txt, data, filename=None): + def renderSegmentPart(self, path, seg_part, data): self._ensureLoaded() + if not _string_needs_render(seg_part.content): + return seg_part.content + + part_path = _make_segment_part_path(path, seg_part.offset) + self.env.loader.segment_parts_cache[part_path] = ( + path, seg_part.content) try: - tpl = self.env.from_string(txt) + tpl = self.env.get_template(part_path) except TemplateSyntaxError as tse: - raise self._getTemplatingError(tse, filename=filename) + raise self._getTemplatingError(tse, filename=path) except TemplateNotFound: raise TemplateNotFoundError() @@ -87,8 +93,8 @@ autoescape = True logger.debug("Creating Jinja environment with folders: %s" % - self.app.templates_dirs) - loader = FileSystemLoader(self.app.templates_dirs) + self.app.templates_dirs) + loader = PieCrustLoader(self.app.templates_dirs) extensions = [ PieCrustHighlightExtension, PieCrustCacheExtension, @@ -102,6 +108,42 @@ extensions=extensions) +def _string_needs_render(txt): + index = txt.find('{') + while index >= 0: + ch = txt[index + 1] + if ch == '{' or ch == '%': + return True + index = txt.find('{', index + 1) + return False + + +def _make_segment_part_path(path, start): + return '$part=%s:%d' % (path, start) + + +class PieCrustLoader(FileSystemLoader): + def __init__(self, searchpath, encoding='utf-8'): + super(PieCrustLoader, self).__init__(searchpath, encoding) + self.segment_parts_cache = {} + + def get_source(self, environment, template): + if template.startswith('$part='): + filename, seg_part = self.segment_parts_cache[template] + + mtime = os.path.getmtime(filename) + + def uptodate(): + try: + return os.path.getmtime(filename) == mtime + except OSError: + return False + + return seg_part, filename, uptodate + + return super(PieCrustLoader, self).get_source(environment, template) + + class PieCrustEnvironment(Environment): def __init__(self, app, *args, **kwargs): self.app = app @@ -301,7 +343,8 @@ args = [parser.parse_expression()] # Extract optional arguments. - kwarg_names = {'line_numbers': 0, 'use_classes': 0, 'class': 1, 'id': 1} + kwarg_names = {'line_numbers': 0, 'use_classes': 0, 'class': 1, + 'id': 1} kwargs = {} while not parser.stream.current.test('block_end'): name = parser.stream.expect('name') @@ -321,7 +364,7 @@ [], [], body).set_lineno(lineno) def _highlight(self, lang, line_numbers=False, use_classes=False, - css_class=None, css_id=None, caller=None): + css_class=None, css_id=None, caller=None): # Try to be mostly compatible with Jinja2-highlight's settings. body = caller() @@ -375,12 +418,12 @@ # now we parse the body of the cache block up to `endpccache` and # drop the needle (which would always be `endpccache` in that case) body = parser.parse_statements(['name:endpccache', 'name:endcache'], - drop_needle=True) + drop_needle=True) # now return a `CallBlock` node that calls our _cache_support # helper method on this extension. return CallBlock(self.call_method('_cache_support', args), - [], [], body).set_lineno(lineno) + [], [], body).set_lineno(lineno) def _cache_support(self, name, caller): key = self.environment.piecrust_cache_prefix + name
--- a/piecrust/templating/pystacheengine.py Thu Jun 25 05:51:47 2015 +1000 +++ b/piecrust/templating/pystacheengine.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,5 +1,7 @@ import logging +import collections.abc import pystache +import pystache.common from piecrust.templating.base import ( TemplateEngine, TemplateNotFoundError, TemplatingError) @@ -14,14 +16,14 @@ def __init__(self): self.renderer = None - def renderString(self, txt, data, filename=None): + def renderSegmentPart(self, path, seg_part, data): self._ensureLoaded() try: - return self.renderer.render(txt, data) - except pystache.TemplateNotFoundError as ex: + return self.renderer.render(seg_part.content, data) + except pystache.common.TemplateNotFoundError as ex: raise TemplateNotFoundError() from ex - except pystache.PystacheError as ex: - raise TemplatingError(str(ex), filename) from ex + except pystache.common.PystacheError as ex: + raise TemplatingError(str(ex), path) from ex def renderFile(self, paths, data): self._ensureLoaded() @@ -45,13 +47,35 @@ try: return self.renderer.render(tpl, data) - except pystache.PystacheError as ex: + except pystache.common.PystacheError as ex: raise TemplatingError(str(ex)) from ex def _ensureLoaded(self): if self.renderer: return - self.renderer = pystache.Renderer( + self.renderer = _WorkaroundRenderer( search_dirs=self.app.templates_dirs) + +_knowns = ['PieCrustData', 'LazyPageConfigData', 'Paginator', 'Assetor', + 'PageLinkerData'] + + +class _WorkaroundRenderer(pystache.Renderer): + def _make_resolve_context(self): + mrc = super(_WorkaroundRenderer, self)._make_resolve_context() + + def _workaround(stack, name): + # Pystache will treat anything that's not a string or a dict as + # a list. This is just plain wrong, but it will take a while before + # the project can get patches on Pypi. + res = mrc(stack, name) + if res is not None and ( + res.__class__.__name__ in _knowns or + isinstance(res, collections.abc.Mapping)): + res = [res] + return res + + return _workaround +
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/piecrust/workerpool.py Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,349 @@ +import os +import sys +import zlib +import logging +import itertools +import threading +import multiprocessing +from piecrust.fastpickle import pickle, unpickle + + +logger = logging.getLogger(__name__) + + +class IWorker(object): + def initialize(self): + raise NotImplementedError() + + def process(self, job): + raise NotImplementedError() + + def getReport(self): + return None + + +TASK_JOB = 0 +TASK_BATCH = 1 +TASK_END = 2 + + +def worker_func(params): + if params.is_profiling: + try: + import cProfile as profile + except ImportError: + import profile + + params.is_profiling = False + name = params.worker_class.__name__ + profile.runctx('_real_worker_func(params)', + globals(), locals(), + filename='%s-%d.prof' % (name, params.wid)) + else: + _real_worker_func(params) + + +def _real_worker_func(params): + wid = params.wid + logger.debug("Worker %d initializing..." % wid) + + params.inqueue._writer.close() + params.outqueue._reader.close() + + w = params.worker_class(*params.initargs) + w.wid = wid + try: + w.initialize() + except Exception as ex: + logger.error("Working failed to initialize:") + logger.exception(ex) + params.outqueue.put(None) + return + + get = params.inqueue.get + put = params.outqueue.put + + completed = 0 + while True: + try: + task = get() + except (EOFError, OSError): + logger.debug("Worker %d encountered connection problem." % wid) + break + + task_type, task_data = task + if task_type == TASK_END: + logger.debug("Worker %d got end task, exiting." % wid) + try: + rep = (task_type, True, wid, (wid, w.getReport())) + except Exception as e: + if params.wrap_exception: + e = multiprocessing.ExceptionWithTraceback( + e, e.__traceback__) + rep = (task_type, False, wid, (wid, e)) + put(rep) + break + + if task_type == TASK_JOB: + task_data = (task_data,) + + for t in task_data: + try: + res = (TASK_JOB, True, wid, w.process(t)) + except Exception as e: + if params.wrap_exception: + e = multiprocessing.ExceptionWithTraceback( + e, e.__traceback__) + res = (TASK_JOB, False, wid, e) + put(res) + + completed += 1 + + logger.debug("Worker %d completed %d tasks." % (wid, completed)) + + +class _WorkerParams(object): + def __init__(self, wid, inqueue, outqueue, worker_class, initargs=(), + wrap_exception=False, is_profiling=False): + self.wid = wid + self.inqueue = inqueue + self.outqueue = outqueue + self.worker_class = worker_class + self.initargs = initargs + self.wrap_exception = wrap_exception + self.is_profiling = is_profiling + + +class WorkerPool(object): + def __init__(self, worker_class, initargs=(), + worker_count=None, batch_size=None, + wrap_exception=False): + worker_count = worker_count or os.cpu_count() or 1 + + use_fastqueue = True + if use_fastqueue: + self._task_queue = FastQueue() + self._result_queue = FastQueue() + self._quick_put = self._task_queue.put + self._quick_get = self._result_queue.get + else: + self._task_queue = multiprocessing.SimpleQueue() + self._result_queue = multiprocessing.SimpleQueue() + self._quick_put = self._task_queue._writer.send + self._quick_get = self._result_queue._reader.recv + + self._batch_size = batch_size + self._callback = None + self._error_callback = None + self._listener = None + + main_module = sys.modules['__main__'] + is_profiling = os.path.basename(main_module.__file__) in [ + 'profile.py', 'cProfile.py'] + + self._pool = [] + for i in range(worker_count): + worker_params = _WorkerParams( + i, self._task_queue, self._result_queue, + worker_class, initargs, + wrap_exception=wrap_exception, + is_profiling=is_profiling) + w = multiprocessing.Process(target=worker_func, + args=(worker_params,)) + w.name = w.name.replace('Process', 'PoolWorker') + w.daemon = True + w.start() + self._pool.append(w) + + self._result_handler = threading.Thread( + target=WorkerPool._handleResults, + args=(self,)) + self._result_handler.daemon = True + self._result_handler.start() + + self._closed = False + + def setHandler(self, callback=None, error_callback=None): + self._callback = callback + self._error_callback = error_callback + + def queueJobs(self, jobs, handler=None, chunk_size=None): + if self._closed: + raise Exception("This worker pool has been closed.") + if self._listener is not None: + raise Exception("A previous job queue has not finished yet.") + + if any([not p.is_alive() for p in self._pool]): + raise Exception("Some workers have prematurely exited.") + + if handler is not None: + self.setHandler(handler) + + if not hasattr(jobs, '__len__'): + jobs = list(jobs) + job_count = len(jobs) + + res = AsyncResult(self, job_count) + if res._count == 0: + res._event.set() + return res + + self._listener = res + + if chunk_size is None: + chunk_size = self._batch_size + if chunk_size is None: + chunk_size = max(1, job_count // 50) + logger.debug("Using chunk size of %d" % chunk_size) + + if chunk_size is None or chunk_size == 1: + for job in jobs: + self._quick_put((TASK_JOB, job)) + else: + it = iter(jobs) + while True: + batch = tuple([i for i in itertools.islice(it, chunk_size)]) + if not batch: + break + self._quick_put((TASK_BATCH, batch)) + + return res + + def close(self): + if self._listener is not None: + raise Exception("A previous job queue has not finished yet.") + + logger.debug("Closing worker pool...") + handler = _ReportHandler(len(self._pool)) + self._callback = handler._handle + for w in self._pool: + self._quick_put((TASK_END, None)) + for w in self._pool: + w.join() + + logger.debug("Waiting for reports...") + if not handler.wait(2): + missing = handler.reports.index(None) + logger.warning( + "Didn't receive all worker reports before timeout. " + "Missing report from worker %d." % missing) + + logger.debug("Exiting result handler thread...") + self._result_queue.put(None) + self._result_handler.join() + self._closed = True + + return handler.reports + + @staticmethod + def _handleResults(pool): + while True: + try: + res = pool._quick_get() + except (EOFError, OSError): + logger.debug("Result handler thread encountered connection " + "problem, exiting.") + return + + if res is None: + logger.debug("Result handler exiting.") + break + + task_type, success, wid, data = res + try: + if success and pool._callback: + pool._callback(data) + elif not success: + if pool._error_callback: + pool._error_callback(data) + else: + logger.error(data) + except Exception as ex: + logger.exception(ex) + + if task_type == TASK_JOB: + pool._listener._onTaskDone() + + +class AsyncResult(object): + def __init__(self, pool, count): + self._pool = pool + self._count = count + self._event = threading.Event() + + def ready(self): + return self._event.is_set() + + def wait(self, timeout=None): + return self._event.wait(timeout) + + def _onTaskDone(self): + self._count -= 1 + if self._count == 0: + self._pool.setHandler(None) + self._pool._listener = None + self._event.set() + + +class _ReportHandler(object): + def __init__(self, worker_count): + self.reports = [None] * worker_count + self._count = worker_count + self._received = 0 + self._event = threading.Event() + + def wait(self, timeout=None): + return self._event.wait(timeout) + + def _handle(self, res): + wid, data = res + if wid < 0 or wid > self._count: + logger.error("Ignoring report from unknown worker %d." % wid) + return + + self._received += 1 + self.reports[wid] = data + + if self._received == self._count: + self._event.set() + + def _handleError(self, res): + wid, data = res + logger.error("Worker %d failed to send its report." % wid) + logger.exception(data) + + +class FastQueue(object): + def __init__(self, compress=False): + self._reader, self._writer = multiprocessing.Pipe(duplex=False) + self._rlock = multiprocessing.Lock() + self._wlock = multiprocessing.Lock() + self._compress = compress + + def __getstate__(self): + return (self._reader, self._writer, self._rlock, self._wlock, + self._compress) + + def __setstate__(self, state): + (self._reader, self._writer, self._rlock, self._wlock, + self._compress) = state + + def get(self): + with self._rlock: + raw = self._reader.recv_bytes() + if self._compress: + data = zlib.decompress(raw) + else: + data = raw + obj = unpickle(data) + return obj + + def put(self, obj): + data = pickle(obj) + if self._compress: + raw = zlib.compress(data) + else: + raw = data + with self._wlock: + self._writer.send_bytes(raw) +
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tests/basefs.py Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,104 @@ +import os.path +import yaml +from piecrust.app import PieCrust + + +class TestFileSystemBase(object): + def __init__(self): + pass + + def _initDefaultSpec(self): + self.withDir('counter') + self.withFile( + 'kitchen/config.yml', + "site:\n title: Mock Website\n") + + def path(self, p): + raise NotImplementedError() + + def getStructure(self, path=None): + raise NotImplementedError() + + def getFileEntry(self, path): + raise NotImplementedError() + + def _createDir(self, path): + raise NotImplementedError() + + def _createFile(self, path, contents): + raise NotImplementedError() + + def getApp(self, cache=True): + root_dir = self.path('/kitchen') + return PieCrust(root_dir, cache=cache, debug=True) + + def withDir(self, path): + path = path.replace('\\', '/') + path = path.lstrip('/') + path = '/%s/%s' % (self._root, path) + self._createDir(path) + return self + + def withFile(self, path, contents): + path = path.replace('\\', '/') + path = path.lstrip('/') + path = '/%s/%s' % (self._root, path) + self._createFile(path, contents) + return self + + def withAsset(self, path, contents): + return self.withFile('kitchen/' + path, contents) + + def withAssetDir(self, path): + return self.withDir('kitchen/' + path) + + def withConfig(self, config): + return self.withFile( + 'kitchen/config.yml', + yaml.dump(config)) + + def withThemeConfig(self, config): + return self.withFile( + 'kitchen/theme/theme_config.yml', + yaml.dump(config)) + + def withPage(self, url, config=None, contents=None): + config = config or {} + contents = contents or "A test page." + text = "---\n" + text += yaml.dump(config) + text += "---\n" + text += contents + + name, ext = os.path.splitext(url) + if not ext: + url += '.md' + url = url.lstrip('/') + return self.withAsset(url, text) + + def withPageAsset(self, page_url, name, contents=None): + contents = contents or "A test asset." + url_base, ext = os.path.splitext(page_url) + dirname = url_base + '-assets' + return self.withAsset( + '%s/%s' % (dirname, name), contents) + + def withPages(self, num, url_factory, config_factory=None, + contents_factory=None): + for i in range(num): + if isinstance(url_factory, str): + url = url_factory.format(idx=i, idx1=(i + 1)) + else: + url = url_factory(i) + + config = None + if config_factory: + config = config_factory(i) + + contents = None + if contents_factory: + contents = contents_factory(i) + + self.withPage(url, config, contents) + return self +
--- a/tests/conftest.py Thu Jun 25 05:51:47 2015 +1000 +++ b/tests/conftest.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,5 +1,6 @@ import io import sys +import time import pprint import os.path import logging @@ -33,6 +34,8 @@ category = os.path.basename(path.dirname) if category == 'bakes': return BakeTestFile(path, parent) + elif category == 'procs': + return PipelineTestFile(path, parent) elif category == 'cli': return ChefTestFile(path, parent) elif category == 'servings': @@ -77,6 +80,37 @@ return fs +def check_expected_outputs(spec, fs, error_type): + cctx = CompareContext() + expected_output_files = spec.get('out') + if expected_output_files: + actual = fs.getStructure('kitchen/_counter') + error = _compare_dicts(expected_output_files, actual, cctx) + if error: + raise error_type(error) + + expected_partial_files = spec.get('outfiles') + if expected_partial_files: + keys = list(sorted(expected_partial_files.keys())) + for key in keys: + try: + actual = fs.getFileEntry('kitchen/_counter/' + + key.lstrip('/')) + except Exception as e: + raise error_type([ + "Can't access output file %s: %s" % (key, e)]) + + expected = expected_partial_files[key] + # HACK because for some reason PyYAML adds a new line for + # those and I have no idea why. + actual = actual.rstrip('\n') + expected = expected.rstrip('\n') + cctx.path = key + cmpres = _compare_str(expected, actual, cctx) + if cmpres: + raise error_type(cmpres) + + class ChefTestItem(YamlTestItemBase): __initialized_logging__ = False @@ -136,11 +170,6 @@ def runtest(self): fs = self._prepareMockFs() - # Output file-system. - expected_output_files = self.spec.get('out') - expected_partial_files = self.spec.get('outfiles') - - # Bake! from piecrust.baking.baker import Baker with mock_fs_scope(fs): out_dir = fs.path('kitchen/_counter') @@ -148,36 +177,13 @@ baker = Baker(app, out_dir) record = baker.bake() - if not record.success: - errors = [] - for e in record.entries: - errors += e.getAllErrors() - raise BakeError(errors) - - if expected_output_files: - actual = fs.getStructure('kitchen/_counter') - error = _compare_dicts(expected_output_files, actual) - if error: - raise ExpectedBakeOutputError(error) + if not record.success: + errors = [] + for e in record.entries: + errors += e.getAllErrors() + raise BakeError(errors) - if expected_partial_files: - keys = list(sorted(expected_partial_files.keys())) - for key in keys: - try: - actual = fs.getFileEntry('kitchen/_counter/' + - key.lstrip('/')) - except Exception as e: - raise ExpectedBakeOutputError([ - "Can't access output file %s: %s" % (key, e)]) - - expected = expected_partial_files[key] - # HACK because for some reason PyYAML adds a new line for those - # and I have no idea why. - actual = actual.rstrip('\n') - expected = expected.rstrip('\n') - cmpres = _compare_str(expected, actual, key) - if cmpres: - raise ExpectedBakeOutputError(cmpres) + check_expected_outputs(self.spec, fs, ExpectedBakeOutputError) def reportinfo(self): return self.fspath, 0, "bake: %s" % self.name @@ -207,6 +213,58 @@ __item_class__ = BakeTestItem +class PipelineTestItem(YamlTestItemBase): + def runtest(self): + fs = self._prepareMockFs() + + from piecrust.processing.pipeline import ProcessorPipeline + with mock_fs_scope(fs): + out_dir = fs.path('kitchen/_counter') + app = fs.getApp() + pipeline = ProcessorPipeline(app, out_dir) + + proc_names = self.spec.get('processors') + if proc_names: + pipeline.enabled_processors = proc_names + + record = pipeline.run() + + if not record.success: + errors = [] + for e in record.entries: + errors += e.errors + raise PipelineError(errors) + + check_expected_outputs(self.spec, fs, ExpectedPipelineOutputError) + + def reportinfo(self): + return self.fspath, 0, "pipeline: %s" % self.name + + def repr_failure(self, excinfo): + if isinstance(excinfo.value, ExpectedPipelineOutputError): + return ('\n'.join( + ['Unexpected pipeline output. Left is expected output, ' + 'right is actual output'] + + excinfo.value.args[0])) + elif isinstance(excinfo.value, PipelineError): + return ('\n'.join( + ['Errors occured during processing:'] + + excinfo.value.args[0])) + return super(PipelineTestItem, self).repr_failure(excinfo) + + +class PipelineError(Exception): + pass + + +class ExpectedPipelineOutputError(Exception): + pass + + +class PipelineTestFile(YamlTestFileBase): + __item_class__ = PipelineTestItem + + class ServeTestItem(YamlTestItemBase): class _TestApp(object): def __init__(self, server): @@ -265,65 +323,86 @@ _add_mock_files(fs, path, subspec) -def _compare(left, right, path): +class CompareContext(object): + def __init__(self, path=None, t=None): + self.path = path or '' + self.time = t or time.time() + + def createChildContext(self, name): + ctx = CompareContext( + path='%s/%s' % (self.path, name), + t=self.time) + return ctx + + +def _compare(left, right, ctx): if type(left) != type(right): return (["Different items: ", - "%s: %s" % (path, pprint.pformat(left)), - "%s: %s" % (path, pprint.pformat(right))]) + "%s: %s" % (ctx.path, pprint.pformat(left)), + "%s: %s" % (ctx.path, pprint.pformat(right))]) if isinstance(left, str): - return _compare_str(left, right, path) + return _compare_str(left, right, ctx) elif isinstance(left, dict): - return _compare_dicts(left, right, path) + return _compare_dicts(left, right, ctx) elif isinstance(left, list): - return _compare_lists(left, right, path) + return _compare_lists(left, right, ctx) elif left != right: return (["Different items: ", - "%s: %s" % (path, pprint.pformat(left)), - "%s: %s" % (path, pprint.pformat(right))]) + "%s: %s" % (ctx.path, pprint.pformat(left)), + "%s: %s" % (ctx.path, pprint.pformat(right))]) -def _compare_dicts(left, right, basepath=''): +def _compare_dicts(left, right, ctx): key_diff = set(left.keys()) ^ set(right.keys()) if key_diff: extra_left = set(left.keys()) - set(right.keys()) if extra_left: return (["Left contains more items: "] + - ['- %s/%s' % (basepath, k) for k in extra_left]) + ['- %s/%s' % (ctx.path, k) for k in extra_left]) extra_right = set(right.keys()) - set(left.keys()) if extra_right: return (["Right contains more items: "] + - ['- %s/%s' % (basepath, k) for k in extra_right]) + ['- %s/%s' % (ctx.path, k) for k in extra_right]) return ["Unknown difference"] for key in left.keys(): lv = left[key] rv = right[key] - childpath = basepath + '/' + key - cmpres = _compare(lv, rv, childpath) + child_ctx = ctx.createChildContext(key) + cmpres = _compare(lv, rv, child_ctx) if cmpres: return cmpres return None -def _compare_lists(left, right, path): +def _compare_lists(left, right, ctx): for i in range(min(len(left), len(right))): l = left[i] r = right[i] - cmpres = _compare(l, r, path) + cmpres = _compare(l, r, ctx) if cmpres: return cmpres if len(left) > len(right): - return (["Left '%s' contains more items. First extra item: " % path, - left[len(right)]]) + return (["Left '%s' contains more items. First extra item: " % + ctx.path, left[len(right)]]) if len(right) > len(left): - return (["Right '%s' contains more items. First extra item: " % path, - right[len(left)]]) + return (["Right '%s' contains more items. First extra item: " % + ctx.path, right[len(left)]]) return None -def _compare_str(left, right, path): +def _compare_str(left, right, ctx): if left == right: return None + + test_time_iso8601 = time.strftime('%Y-%m-%dT%H:%M:%SZ', + time.gmtime(ctx.time)) + replacements = { + '%test_time_iso8601%': test_time_iso8601} + for token, repl in replacements.items(): + left = left.replace(token, repl) + right = right.replace(token, repl) + for i in range(min(len(left), len(right))): if left[i] != right[i]: start = max(0, i - 15) @@ -346,7 +425,7 @@ if j < i: r_offset += len(c) - return ["Items '%s' differ at index %d:" % (path, i), '', + return ["Items '%s' differ at index %d:" % (ctx.path, i), '', "Left:", left, '', "Right:", right, '', "Difference:", @@ -354,12 +433,12 @@ r_str, (' ' * r_offset + '^')] if len(left) > len(right): return ["Left is longer.", - "Left '%s': " % path, left, - "Right '%s': " % path, right, + "Left '%s': " % ctx.path, left, + "Right '%s': " % ctx.path, right, "Extra items: %r" % left[len(right):]] if len(right) > len(left): return ["Right is longer.", - "Left '%s': " % path, left, - "Right '%s': " % path, right, + "Left '%s': " % ctx.path, left, + "Right '%s': " % ctx.path, right, "Extra items: %r" % right[len(left):]]
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tests/memfs.py Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,309 @@ +import os.path +import io +import time +import errno +import random +import codecs +import shutil +import mock +from piecrust import RESOURCES_DIR +from .basefs import TestFileSystemBase + + +class _MockFsEntry(object): + def __init__(self, contents): + self.contents = contents + self.metadata = {'mtime': time.time()} + + +class _MockFsEntryWriter(object): + def __init__(self, entry, mode='rt'): + self._entry = entry + self._mode = mode + + if 'b' in mode: + data = entry.contents + if isinstance(data, str): + data = data.encode('utf8') + self._stream = io.BytesIO(data) + else: + self._stream = io.StringIO(entry.contents) + + def __getattr__(self, name): + return getattr(self._stream, name) + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc_value, exc_tb): + if 'w' in self._mode: + if 'a' in self._mode: + self._entry.contents += self._stream.getvalue() + else: + self._entry.contents = self._stream.getvalue() + self._entry.metadata['mtime'] = time.time() + self._stream.close() + + +class MemoryFileSystem(TestFileSystemBase): + def __init__(self, default_spec=True): + self._root = 'root_%d' % random.randrange(1000) + self._fs = {self._root: {}} + if default_spec: + self._initDefaultSpec() + + def path(self, p): + p = p.replace('\\', '/') + if p in ['/', '', None]: + return '/%s' % self._root + return '/%s/%s' % (self._root, p.lstrip('/')) + + def getStructure(self, path=None): + root = self._fs[self._root] + if path: + root = self._getEntry(self.path(path)) + if root is None: + raise Exception("No such path: %s" % path) + if not isinstance(root, dict): + raise Exception("Path is not a directory: %s" % path) + + res = {} + for k, v in root.items(): + self._getStructureRecursive(v, res, k) + return res + + def getFileEntry(self, path): + entry = self._getEntry(self.path(path)) + if entry is None: + raise Exception("No such file: %s" % path) + if not isinstance(entry, _MockFsEntry): + raise Exception("Path is not a file: %s" % path) + return entry.contents + + def _getStructureRecursive(self, src, target, name): + if isinstance(src, _MockFsEntry): + target[name] = src.contents + return + + e = {} + for k, v in src.items(): + self._getStructureRecursive(v, e, k) + target[name] = e + + def _getEntry(self, path): + cur = self._fs + path = path.replace('\\', '/').lstrip('/') + bits = path.split('/') + for p in bits: + try: + cur = cur[p] + except KeyError: + return None + return cur + + def _createDir(self, path): + cur = self._fs + path = path.replace('\\', '/').strip('/') + bits = path.split('/') + for b in bits: + if b not in cur: + cur[b] = {} + cur = cur[b] + return self + + def _createFile(self, path, contents): + cur = self._fs + path = path.replace('\\', '/').lstrip('/') + bits = path.split('/') + for b in bits[:-1]: + if b not in cur: + cur[b] = {} + cur = cur[b] + cur[bits[-1]] = _MockFsEntry(contents) + return self + + def _deleteEntry(self, path): + parent = self._getEntry(os.path.dirname(path)) + assert parent is not None + name = os.path.basename(path) + assert name in parent + del parent[name] + + +class MemoryScope(object): + def __init__(self, fs, open_patches=None): + self.open_patches = open_patches or [] + self._fs = fs + self._patchers = [] + self._originals = {} + + @property + def root(self): + return self._fs._root + + def __enter__(self): + self._startMock() + return self + + def __exit__(self, type, value, traceback): + self._endMock() + + def _startMock(self): + # TODO: sadly, there seems to be no way to replace `open` everywhere? + modules = self.open_patches + [ + '__main__', + 'piecrust.records', + 'jinja2.utils'] + for m in modules: + self._createMock('%s.open' % m, open, self._open, create=True) + + self._createMock('codecs.open', codecs.open, self._codecsOpen) + self._createMock('os.listdir', os.listdir, self._listdir) + self._createMock('os.makedirs', os.makedirs, self._makedirs) + self._createMock('os.remove', os.remove, self._remove) + self._createMock('os.rename', os.rename, self._rename) + self._createMock('os.path.exists', os.path.exists, self._exists) + self._createMock('os.path.isdir', os.path.isdir, self._isdir) + self._createMock('os.path.isfile', os.path.isfile, self._isfile) + self._createMock('os.path.islink', os.path.islink, self._islink) + self._createMock('os.path.getmtime', os.path.getmtime, self._getmtime) + self._createMock('shutil.copyfile', shutil.copyfile, self._copyfile) + self._createMock('shutil.rmtree', shutil.rmtree, self._rmtree) + for p in self._patchers: + p.start() + + def _endMock(self): + for p in self._patchers: + p.stop() + + def _createMock(self, name, orig, func, **kwargs): + self._originals[name] = orig + self._patchers.append(mock.patch(name, func, **kwargs)) + + def _doOpen(self, orig_name, path, mode, *args, **kwargs): + path = os.path.normpath(path) + if path.startswith(RESOURCES_DIR): + return self._originals[orig_name](path, mode, *args, **kwargs) + + if 'r' in mode: + e = self._getFsEntry(path) + elif 'w' in mode or 'x' in mode or 'a' in mode: + e = self._getFsEntry(path) + if e is None: + contents = '' + if 'b' in mode: + contents = bytes() + self._fs._createFile(path, contents) + e = self._getFsEntry(path) + assert e is not None + elif 'x' in mode: + err = IOError("File '%s' already exists" % path) + err.errno = errno.EEXIST + raise err + else: + err = IOError("Unsupported open mode: %s" % mode) + err.errno = errno.EINVAL + raise err + + if e is None: + err = IOError("No such file: %s" % path) + err.errno = errno.ENOENT + raise err + if not isinstance(e, _MockFsEntry): + err = IOError("'%s' is not a file %s" % (path, e)) + err.errno = errno.EISDIR + raise err + + return _MockFsEntryWriter(e, mode) + + def _open(self, path, mode, *args, **kwargs): + return self._doOpen('__main__.open', path, mode, *args, **kwargs) + + def _codecsOpen(self, path, mode, *args, **kwargs): + return self._doOpen('codecs.open', path, mode, *args, **kwargs) + + def _listdir(self, path): + path = os.path.normpath(path) + if path.startswith(RESOURCES_DIR): + return self._originals['os.listdir'](path) + + e = self._getFsEntry(path) + if e is None: + raise OSError("No such directory: %s" % path) + if not isinstance(e, dict): + raise OSError("'%s' is not a directory." % path) + return list(e.keys()) + + def _makedirs(self, path, mode=0o777): + if not path.replace('\\', '/').startswith('/' + self.root): + raise Exception("Shouldn't create directory: %s" % path) + self._fs._createDir(path) + + def _remove(self, path): + path = os.path.normpath(path) + self._fs._deleteEntry(path) + + def _exists(self, path): + path = os.path.normpath(path) + if path.startswith(RESOURCES_DIR): + return self._originals['os.path.isdir'](path) + e = self._getFsEntry(path) + return e is not None + + def _isdir(self, path): + path = os.path.normpath(path) + if path.startswith(RESOURCES_DIR): + return self._originals['os.path.isdir'](path) + e = self._getFsEntry(path) + return e is not None and isinstance(e, dict) + + def _isfile(self, path): + path = os.path.normpath(path) + if path.startswith(RESOURCES_DIR): + return self._originals['os.path.isfile'](path) + e = self._getFsEntry(path) + return e is not None and isinstance(e, _MockFsEntry) + + def _islink(self, path): + path = os.path.normpath(path) + if path.startswith(RESOURCES_DIR): + return self._originals['os.path.islink'](path) + return False + + def _getmtime(self, path): + path = os.path.normpath(path) + if path.startswith(RESOURCES_DIR): + return self._originals['os.path.getmtime'](path) + e = self._getFsEntry(path) + if e is None: + raise OSError("No such file: %s" % path) + return e.metadata['mtime'] + + def _copyfile(self, src, dst): + src = os.path.normpath(src) + if src.startswith(RESOURCES_DIR): + with self._originals['__main__.open'](src, 'r') as fp: + src_text = fp.read() + else: + e = self._getFsEntry(src) + src_text = e.contents + if not dst.replace('\\', '/').startswith('/' + self.root): + raise Exception("Shouldn't copy to: %s" % dst) + self._fs._createFile(dst, src_text) + + def _rename(self, src, dst): + src = os.path.normpath(src) + if src.startswith(RESOURCES_DIR) or dst.startswith(RESOURCES_DIR): + raise Exception("Shouldn't rename files in the resources path.") + self._copyfile(src, dst) + self._remove(src) + + def _rmtree(self, path): + if not path.replace('\\', '/').startswith('/' + self.root): + raise Exception("Shouldn't delete trees from: %s" % path) + e = self._fs._getEntry(os.path.dirname(path)) + del e[os.path.basename(path)] + + def _getFsEntry(self, path): + return self._fs._getEntry(path) +
--- a/tests/mockutil.py Thu Jun 25 05:51:47 2015 +1000 +++ b/tests/mockutil.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,23 +1,10 @@ -import io -import time -import errno -import random -import codecs -import shutil import os.path import mock -import yaml from piecrust.app import PieCrust, PieCrustConfiguration from piecrust.page import Page from piecrust.rendering import QualifiedPage, PageRenderingContext, render_page -resources_path = os.path.abspath( - os.path.join( - os.path.dirname(__file__), - '..', 'piecrust', 'resources')) - - def get_mock_app(config=None): app = mock.MagicMock(spec=PieCrust) app.config = PieCrustConfiguration() @@ -37,376 +24,7 @@ return rp.content -class _MockFsEntry(object): - def __init__(self, contents): - self.contents = contents - self.metadata = {'mtime': time.time()} - - -class _MockFsEntryWriter(object): - def __init__(self, entry, mode='rt'): - self._entry = entry - self._mode = mode - - if 'b' in mode: - data = entry.contents - if isinstance(data, str): - data = data.encode('utf8') - self._stream = io.BytesIO(data) - else: - self._stream = io.StringIO(entry.contents) - - def __getattr__(self, name): - return getattr(self._stream, name) - - def __enter__(self): - return self - - def __exit__(self, exc_type, exc_value, exc_tb): - if 'w' in self._mode: - if 'a' in self._mode: - self._entry.contents += self._stream.getvalue() - else: - self._entry.contents = self._stream.getvalue() - self._entry.metadata['mtime'] = time.time() - self._stream.close() - - -class mock_fs(object): - def __init__(self, default_spec=True): - self._root = 'root_%d' % random.randrange(1000) - self._fs = {self._root: {}} - if default_spec: - self.withDir('counter') - self.withFile('kitchen/config.yml', - "site:\n title: Mock Website\n") - - def path(self, p): - p = p.replace('\\', '/') - if p in ['/', '', None]: - return '/%s' % self._root - return '/%s/%s' % (self._root, p.lstrip('/')) - - def getApp(self, cache=True): - root_dir = self.path('/kitchen') - return PieCrust(root_dir, cache=cache, debug=True) - - def withDir(self, path): - path = path.replace('\\', '/') - path = path.lstrip('/') - path = '/%s/%s' % (self._root, path) - self._createDir(path) - return self - - def withFile(self, path, contents): - path = path.replace('\\', '/') - path = path.lstrip('/') - path = '/%s/%s' % (self._root, path) - self._createFile(path, contents) - return self - - def withAsset(self, path, contents): - return self.withFile('kitchen/' + path, contents) - - def withAssetDir(self, path): - return self.withDir('kitchen/' + path) - - def withConfig(self, config): - return self.withFile( - 'kitchen/config.yml', - yaml.dump(config)) - - def withThemeConfig(self, config): - return self.withFile( - 'kitchen/theme/theme_config.yml', - yaml.dump(config)) - - def withPage(self, url, config=None, contents=None): - config = config or {} - contents = contents or "A test page." - text = "---\n" - text += yaml.dump(config) - text += "---\n" - text += contents - - name, ext = os.path.splitext(url) - if not ext: - url += '.md' - url = url.lstrip('/') - return self.withAsset(url, text) - - def withPageAsset(self, page_url, name, contents=None): - contents = contents or "A test asset." - url_base, ext = os.path.splitext(page_url) - dirname = url_base + '-assets' - return self.withAsset('%s/%s' % (dirname, name), - contents) - - def withPages(self, num, url_factory, config_factory=None, - contents_factory=None): - for i in range(num): - if isinstance(url_factory, str): - url = url_factory.format(idx=i, idx1=(i + 1)) - else: - url = url_factory(i) - - config = None - if config_factory: - config = config_factory(i) - - contents = None - if contents_factory: - contents = contents_factory(i) - - self.withPage(url, config, contents) - return self - - def getStructure(self, path=None): - root = self._fs[self._root] - if path: - root = self._getEntry(self.path(path)) - if root is None: - raise Exception("No such path: %s" % path) - if not isinstance(root, dict): - raise Exception("Path is not a directory: %s" % path) - - res = {} - for k, v in root.items(): - self._getStructureRecursive(v, res, k) - return res - - def getFileEntry(self, path): - entry = self._getEntry(self.path(path)) - if entry is None: - raise Exception("No such file: %s" % path) - if not isinstance(entry, _MockFsEntry): - raise Exception("Path is not a file: %s" % path) - return entry.contents - - def _getStructureRecursive(self, src, target, name): - if isinstance(src, _MockFsEntry): - target[name] = src.contents - return - - e = {} - for k, v in src.items(): - self._getStructureRecursive(v, e, k) - target[name] = e - - def _getEntry(self, path): - cur = self._fs - path = path.replace('\\', '/').lstrip('/') - bits = path.split('/') - for p in bits: - try: - cur = cur[p] - except KeyError: - return None - return cur - - def _createDir(self, path): - cur = self._fs - path = path.replace('\\', '/').strip('/') - bits = path.split('/') - for b in bits: - if b not in cur: - cur[b] = {} - cur = cur[b] - return self +from .tmpfs import ( + TempDirFileSystem as mock_fs, + TempDirScope as mock_fs_scope) - def _createFile(self, path, contents): - cur = self._fs - path = path.replace('\\', '/').lstrip('/') - bits = path.split('/') - for b in bits[:-1]: - if b not in cur: - cur[b] = {} - cur = cur[b] - cur[bits[-1]] = _MockFsEntry(contents) - return self - - def _deleteEntry(self, path): - parent = self._getEntry(os.path.dirname(path)) - assert parent is not None - name = os.path.basename(path) - assert name in parent - del parent[name] - - -class mock_fs_scope(object): - def __init__(self, fs, open_patches=None): - self.open_patches = open_patches or [] - self._fs = fs - self._patchers = [] - self._originals = {} - - @property - def root(self): - return self._fs._root - - def __enter__(self): - self._startMock() - return self - - def __exit__(self, type, value, traceback): - self._endMock() - - def _startMock(self): - # TODO: sadly, there seems to be no way to replace `open` everywhere? - modules = self.open_patches + [ - '__main__', - 'piecrust.records', - 'jinja2.utils'] - for m in modules: - self._createMock('%s.open' % m, open, self._open, create=True) - - self._createMock('codecs.open', codecs.open, self._codecsOpen) - self._createMock('os.listdir', os.listdir, self._listdir) - self._createMock('os.makedirs', os.makedirs, self._makedirs) - self._createMock('os.remove', os.remove, self._remove) - self._createMock('os.rename', os.rename, self._rename) - self._createMock('os.path.exists', os.path.exists, self._exists) - self._createMock('os.path.isdir', os.path.isdir, self._isdir) - self._createMock('os.path.isfile', os.path.isfile, self._isfile) - self._createMock('os.path.islink', os.path.islink, self._islink) - self._createMock('os.path.getmtime', os.path.getmtime, self._getmtime) - self._createMock('shutil.copyfile', shutil.copyfile, self._copyfile) - self._createMock('shutil.rmtree', shutil.rmtree, self._rmtree) - for p in self._patchers: - p.start() - - def _endMock(self): - for p in self._patchers: - p.stop() - - def _createMock(self, name, orig, func, **kwargs): - self._originals[name] = orig - self._patchers.append(mock.patch(name, func, **kwargs)) - - def _doOpen(self, orig_name, path, mode, *args, **kwargs): - path = os.path.normpath(path) - if path.startswith(resources_path): - return self._originals[orig_name](path, mode, *args, **kwargs) - - if 'r' in mode: - e = self._getFsEntry(path) - elif 'w' in mode or 'x' in mode or 'a' in mode: - e = self._getFsEntry(path) - if e is None: - contents = '' - if 'b' in mode: - contents = bytes() - self._fs._createFile(path, contents) - e = self._getFsEntry(path) - assert e is not None - elif 'x' in mode: - err = IOError("File '%s' already exists" % path) - err.errno = errno.EEXIST - raise err - else: - err = IOError("Unsupported open mode: %s" % mode) - err.errno = errno.EINVAL - raise err - - if e is None: - err = IOError("No such file: %s" % path) - err.errno = errno.ENOENT - raise err - if not isinstance(e, _MockFsEntry): - err = IOError("'%s' is not a file %s" % (path, e)) - err.errno = errno.EISDIR - raise err - - return _MockFsEntryWriter(e, mode) - - def _open(self, path, mode, *args, **kwargs): - return self._doOpen('__main__.open', path, mode, *args, **kwargs) - - def _codecsOpen(self, path, mode, *args, **kwargs): - return self._doOpen('codecs.open', path, mode, *args, **kwargs) - - def _listdir(self, path): - path = os.path.normpath(path) - if path.startswith(resources_path): - return self._originals['os.listdir'](path) - - e = self._getFsEntry(path) - if e is None: - raise OSError("No such directory: %s" % path) - if not isinstance(e, dict): - raise OSError("'%s' is not a directory." % path) - return list(e.keys()) - - def _makedirs(self, path, mode=0o777): - if not path.replace('\\', '/').startswith('/' + self.root): - raise Exception("Shouldn't create directory: %s" % path) - self._fs._createDir(path) - - def _remove(self, path): - path = os.path.normpath(path) - self._fs._deleteEntry(path) - - def _exists(self, path): - path = os.path.normpath(path) - if path.startswith(resources_path): - return self._originals['os.path.isdir'](path) - e = self._getFsEntry(path) - return e is not None - - def _isdir(self, path): - path = os.path.normpath(path) - if path.startswith(resources_path): - return self._originals['os.path.isdir'](path) - e = self._getFsEntry(path) - return e is not None and isinstance(e, dict) - - def _isfile(self, path): - path = os.path.normpath(path) - if path.startswith(resources_path): - return self._originals['os.path.isfile'](path) - e = self._getFsEntry(path) - return e is not None and isinstance(e, _MockFsEntry) - - def _islink(self, path): - path = os.path.normpath(path) - if path.startswith(resources_path): - return self._originals['os.path.islink'](path) - return False - - def _getmtime(self, path): - path = os.path.normpath(path) - if path.startswith(resources_path): - return self._originals['os.path.getmtime'](path) - e = self._getFsEntry(path) - if e is None: - raise OSError("No such file: %s" % path) - return e.metadata['mtime'] - - def _copyfile(self, src, dst): - src = os.path.normpath(src) - if src.startswith(resources_path): - with self._originals['__main__.open'](src, 'r') as fp: - src_text = fp.read() - else: - e = self._getFsEntry(src) - src_text = e.contents - if not dst.replace('\\', '/').startswith('/' + self.root): - raise Exception("Shouldn't copy to: %s" % dst) - self._fs._createFile(dst, src_text) - - def _rename(self, src, dst): - src = os.path.normpath(src) - if src.startswith(resources_path) or dst.startswith(resources_path): - raise Exception("Shouldn't rename files in the resources path.") - self._copyfile(src, dst) - self._remove(src) - - def _rmtree(self, path): - if not path.replace('\\', '/').startswith('/' + self.root): - raise Exception("Shouldn't delete trees from: %s" % path) - e = self._fs._getEntry(os.path.dirname(path)) - del e[os.path.basename(path)] - - def _getFsEntry(self, path): - return self._fs._getEntry(path) -
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tests/procs/test_sitemap.yaml Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,40 @@ +--- +in: + assets/sitemap.sitemap: | + autogen: [pages, theme_pages] + pages/foo.md: This is a foo +outfiles: + sitemap.xml: | + <?xml version="1.0" encoding="utf-8"?> + <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> + <url> + <loc>/foo.html</loc> + <lastmod>%test_time_iso8601%</lastmod> + </url> + <url> + <loc>/</loc> + <lastmod>%test_time_iso8601%</lastmod> + </url> + </urlset> +--- +in: + assets/sitemap.sitemap: | + autogen: [pages] + pages/foo.md: | + --- + sitemap: + changefreq: monthly + priority: 0.8 + --- + This is a foo +outfiles: + sitemap.xml: | + <?xml version="1.0" encoding="utf-8"?> + <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> + <url> + <loc>/foo.html</loc> + <lastmod>%test_time_iso8601%</lastmod> + <changefreq>monthly</changefreq> + <priority>0.8</priority> + </url> + </urlset>
--- a/tests/test_baking_baker.py Thu Jun 25 05:51:47 2015 +1000 +++ b/tests/test_baking_baker.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,7 +1,8 @@ import time import os.path import pytest -from piecrust.baking.baker import PageBaker, Baker +from piecrust.baking.baker import Baker +from piecrust.baking.single import PageBaker from piecrust.baking.records import BakeRecord from .mockutil import get_mock_app, mock_fs, mock_fs_scope @@ -56,6 +57,7 @@ with mock_fs_scope(fs): out_dir = fs.path('kitchen/_counter') app = fs.getApp() + app.config.set('baker/workers', 1) baker = Baker(app, out_dir) baker.bake() structure = fs.getStructure('kitchen/_counter')
--- a/tests/test_configuration.py Thu Jun 25 05:51:47 2015 +1000 +++ b/tests/test_configuration.py Sat Jul 11 20:33:55 2015 -0700 @@ -13,13 +13,13 @@ ]) def test_config_init(values, expected): config = Configuration(values) - assert config.get() == expected + assert config.getAll() == expected def test_config_set_all(): config = Configuration() config.setAll({'foo': 'bar'}) - assert config.get() == {'foo': 'bar'} + assert config.getAll() == {'foo': 'bar'} def test_config_get_and_set(): @@ -125,7 +125,7 @@ 'child10': 'ten' } } - assert config.get() == expected + assert config.getAll() == expected def test_ordered_loader():
--- a/tests/test_data_assetor.py Thu Jun 25 05:51:47 2015 +1000 +++ b/tests/test_data_assetor.py Sat Jul 11 20:33:55 2015 -0700 @@ -5,34 +5,35 @@ from .mockutil import mock_fs, mock_fs_scope -@pytest.mark.parametrize('fs, site_root, expected', [ - (mock_fs().withPage('pages/foo/bar'), '/', {}), - (mock_fs() +@pytest.mark.parametrize('fs_fac, site_root, expected', [ + (lambda: mock_fs().withPage('pages/foo/bar'), '/', {}), + (lambda: mock_fs() .withPage('pages/foo/bar') .withPageAsset('pages/foo/bar', 'one.txt', 'one'), '/', {'one': 'one'}), - (mock_fs() + (lambda: mock_fs() .withPage('pages/foo/bar') .withPageAsset('pages/foo/bar', 'one.txt', 'one') .withPageAsset('pages/foo/bar', 'two.txt', 'two'), '/', {'one': 'one', 'two': 'two'}), - (mock_fs().withPage('pages/foo/bar'), '/whatever', {}), - (mock_fs() + (lambda: mock_fs().withPage('pages/foo/bar'), '/whatever', {}), + (lambda: mock_fs() .withPage('pages/foo/bar') .withPageAsset('pages/foo/bar', 'one.txt', 'one'), '/whatever', {'one': 'one'}), - (mock_fs() + (lambda: mock_fs() .withPage('pages/foo/bar') .withPageAsset('pages/foo/bar', 'one.txt', 'one') .withPageAsset('pages/foo/bar', 'two.txt', 'two'), '/whatever', {'one': 'one', 'two': 'two'}) ]) -def test_assets(fs, site_root, expected): +def test_assets(fs_fac, site_root, expected): + fs = fs_fac() fs.withConfig({'site': {'root': site_root}}) with mock_fs_scope(fs): page = MagicMock()
--- a/tests/test_data_linker.py Thu Jun 25 05:51:47 2015 +1000 +++ b/tests/test_data_linker.py Sat Jul 11 20:33:55 2015 -0700 @@ -5,17 +5,17 @@ @pytest.mark.parametrize( - 'fs, page_path, expected', + 'fs_fac, page_path, expected', [ - (mock_fs().withPage('pages/foo'), 'foo.md', + (lambda: mock_fs().withPage('pages/foo'), 'foo.md', # is_dir, name, is_self, data [(False, 'foo', True, '/foo')]), - ((mock_fs() + ((lambda: mock_fs() .withPage('pages/foo') .withPage('pages/bar')), 'foo.md', [(False, 'bar', False, '/bar'), (False, 'foo', True, '/foo')]), - ((mock_fs() + ((lambda: mock_fs() .withPage('pages/baz') .withPage('pages/something') .withPage('pages/something/else') @@ -26,7 +26,7 @@ (False, 'baz', False, '/baz'), (False, 'foo', True, '/foo'), (True, 'something', False, '/something')]), - ((mock_fs() + ((lambda: mock_fs() .withPage('pages/something/else') .withPage('pages/foo') .withPage('pages/something/good') @@ -35,7 +35,8 @@ [(False, 'else', True, '/something/else'), (False, 'good', False, '/something/good')]) ]) -def test_linker_iteration(fs, page_path, expected): +def test_linker_iteration(fs_fac, page_path, expected): + fs = fs_fac() with mock_fs_scope(fs): app = fs.getApp() app.config.set('site/pretty_urls', True) @@ -54,16 +55,16 @@ @pytest.mark.parametrize( - 'fs, page_path, expected', + 'fs_fac, page_path, expected', [ - (mock_fs().withPage('pages/foo'), 'foo.md', + (lambda: mock_fs().withPage('pages/foo'), 'foo.md', [('/foo', True)]), - ((mock_fs() + ((lambda: mock_fs() .withPage('pages/foo') .withPage('pages/bar')), 'foo.md', [('/bar', False), ('/foo', True)]), - ((mock_fs() + ((lambda: mock_fs() .withPage('pages/baz') .withPage('pages/something/else') .withPage('pages/foo') @@ -71,7 +72,7 @@ 'foo.md', [('/bar', False), ('/baz', False), ('/foo', True), ('/something/else', False)]), - ((mock_fs() + ((lambda: mock_fs() .withPage('pages/something/else') .withPage('pages/foo') .withPage('pages/something/good') @@ -80,7 +81,8 @@ [('/something/else', True), ('/something/good', False)]) ]) -def test_recursive_linker_iteration(fs, page_path, expected): +def test_recursive_linker_iteration(fs_fac, page_path, expected): + fs = fs_fac() with mock_fs_scope(fs): app = fs.getApp() app.config.set('site/pretty_urls', True)
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tests/test_fastpickle.py Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,53 @@ +import datetime +import pytest +from piecrust.fastpickle import pickle, unpickle + + +class Foo(object): + def __init__(self, name): + self.name = name + self.bars = [] + + +class Bar(object): + def __init__(self, value): + self.value = value + + +@pytest.mark.parametrize( + 'obj, expected', + [ + (True, True), + (42, 42), + (3.14, 3.14), + (datetime.date(2015, 5, 21), datetime.date(2015, 5, 21)), + (datetime.datetime(2015, 5, 21, 12, 55, 32), + datetime.datetime(2015, 5, 21, 12, 55, 32)), + (datetime.time(9, 25, 57), datetime.time(9, 25, 57)), + ((1, 2, 3), (1, 2, 3)), + ([1, 2, 3], [1, 2, 3]), + ({'foo': 1, 'bar': 2}, {'foo': 1, 'bar': 2}), + (set([1, 2, 3]), set([1, 2, 3])), + ({'foo': [1, 2, 3], 'bar': {'one': 1, 'two': 2}}, + {'foo': [1, 2, 3], 'bar': {'one': 1, 'two': 2}}) + ]) +def test_pickle_unpickle(obj, expected): + data = pickle(obj) + actual = unpickle(data) + assert actual == expected + + +def test_objects(): + f = Foo('foo') + f.bars.append(Bar(1)) + f.bars.append(Bar(2)) + + data = pickle(f) + o = unpickle(data) + + assert type(o) == Foo + assert o.name == 'foo' + assert len(o.bars) == 2 + for i in range(2): + assert f.bars[i].value == o.bars[i].value +
--- a/tests/test_processing_base.py Thu Jun 25 05:51:47 2015 +1000 +++ b/tests/test_processing_base.py Sat Jul 11 20:33:55 2015 -0700 @@ -2,8 +2,10 @@ import os.path import shutil import pytest -from piecrust.processing.base import (ProcessorPipeline, SimpleFileProcessor) +from piecrust.processing.base import SimpleFileProcessor +from piecrust.processing.pipeline import ProcessorPipeline from piecrust.processing.records import ProcessorPipelineRecord +from piecrust.processing.worker import get_filtered_processors from .mockutil import mock_fs, mock_fs_scope @@ -36,7 +38,6 @@ def _get_pipeline(fs, app=None): app = app or fs.getApp() - app.config.set('baker/num_workers', 1) return ProcessorPipeline(app, fs.path('counter')) @@ -44,7 +45,7 @@ fs = mock_fs() with mock_fs_scope(fs): pp = _get_pipeline(fs) - pp.filterProcessors(['copy']) + pp.enabled_processors = ['copy'] expected = {} assert expected == fs.getStructure('counter') pp.run() @@ -57,7 +58,7 @@ .withFile('kitchen/assets/something.html', 'A test file.')) with mock_fs_scope(fs): pp = _get_pipeline(fs) - pp.filterProcessors(['copy']) + pp.enabled_processors = ['copy'] expected = {} assert expected == fs.getStructure('counter') pp.run() @@ -70,17 +71,19 @@ .withFile('kitchen/assets/blah.foo', 'A test file.')) with mock_fs_scope(fs): pp = _get_pipeline(fs) - pp.filterProcessors(['copy']) + pp.enabled_processors = ['copy'] pp.run() expected = {'blah.foo': 'A test file.'} assert expected == fs.getStructure('counter') mtime = os.path.getmtime(fs.path('/counter/blah.foo')) assert abs(time.time() - mtime) <= 2 + time.sleep(1) pp.run() assert expected == fs.getStructure('counter') assert mtime == os.path.getmtime(fs.path('/counter/blah.foo')) + time.sleep(1) fs.withFile('kitchen/assets/blah.foo', 'A new test file.') pp.run() expected = {'blah.foo': 'A new test file.'} @@ -91,20 +94,22 @@ def test_two_levels_dirtyness(): fs = (mock_fs() .withFile('kitchen/assets/blah.foo', 'A test file.')) - with mock_fs_scope(fs) as scope: + with mock_fs_scope(fs): pp = _get_pipeline(fs) - pp.processors.append(FooProcessor(('foo', 'bar'), scope._open)) - pp.filterProcessors(['foo', 'copy']) + pp.enabled_processors = ['copy'] + pp.additional_processors = [FooProcessor(('foo', 'bar'))] pp.run() expected = {'blah.bar': 'FOO: A test file.'} assert expected == fs.getStructure('counter') mtime = os.path.getmtime(fs.path('/counter/blah.bar')) assert abs(time.time() - mtime) <= 2 + time.sleep(1) pp.run() assert expected == fs.getStructure('counter') assert mtime == os.path.getmtime(fs.path('/counter/blah.bar')) + time.sleep(1) fs.withFile('kitchen/assets/blah.foo', 'A new test file.') pp.run() expected = {'blah.bar': 'FOO: A new test file.'} @@ -122,10 +127,11 @@ 'blah2.foo': 'Ooops'} assert expected == fs.getStructure('kitchen/assets') pp = _get_pipeline(fs) - pp.filterProcessors(['copy']) + pp.enabled_processors = ['copy'] pp.run() assert expected == fs.getStructure('counter') + time.sleep(1) os.remove(fs.path('/kitchen/assets/blah2.foo')) expected = { 'blah1.foo': 'A test file.'} @@ -140,18 +146,21 @@ with mock_fs_scope(fs): pp = _get_pipeline(fs) noop = NoopProcessor(('foo', 'foo')) - pp.processors.append(noop) - pp.filterProcessors(['foo', 'copy']) + pp.enabled_processors = ['copy'] + pp.additional_processors = [noop] pp.run() - assert 1 == len(noop.processed) + assert os.path.exists(fs.path('/counter/blah.foo')) is True + mtime = os.path.getmtime(fs.path('/counter/blah.foo')) + time.sleep(1) pp.run() - assert 1 == len(noop.processed) + assert mtime == os.path.getmtime(fs.path('/counter/blah.foo')) + time.sleep(1) ProcessorPipelineRecord.RECORD_VERSION += 1 try: pp.run() - assert 2 == len(noop.processed) + assert mtime < os.path.getmtime(fs.path('/counter/blah.foo')) finally: ProcessorPipelineRecord.RECORD_VERSION -= 1 @@ -165,24 +174,27 @@ {'something.html': 'A test file.', 'foo': {'_important.html': 'Important!'}}) ]) -def test_skip_pattern(patterns, expected): +def test_ignore_pattern(patterns, expected): fs = (mock_fs() .withFile('kitchen/assets/something.html', 'A test file.') .withFile('kitchen/assets/_hidden.html', 'Shhh') .withFile('kitchen/assets/foo/_important.html', 'Important!')) with mock_fs_scope(fs): pp = _get_pipeline(fs) - pp.addSkipPatterns(patterns) - pp.filterProcessors(['copy']) + pp.addIgnorePatterns(patterns) + pp.enabled_processors = ['copy'] assert {} == fs.getStructure('counter') pp.run() assert expected == fs.getStructure('counter') @pytest.mark.parametrize('names, expected', [ - ('all', ['copy', 'concat', 'less', 'sass', 'sitemap']), - ('all -sitemap', ['copy', 'concat', 'less', 'sass']), - ('-sitemap -less -sass all', ['copy', 'concat']), + ('all', ['cleancss', 'compass', 'copy', 'concat', 'less', 'requirejs', + 'sass', 'sitemap', 'uglifyjs']), + ('all -sitemap', ['cleancss', 'copy', 'compass', 'concat', 'less', + 'requirejs', 'sass', 'uglifyjs']), + ('-sitemap -less -sass all', ['cleancss', 'copy', 'compass', 'concat', + 'requirejs', 'uglifyjs']), ('copy', ['copy']), ('less sass', ['less', 'sass']) ]) @@ -190,9 +202,8 @@ fs = mock_fs() with mock_fs_scope(fs): app = fs.getApp() - pp = _get_pipeline(fs, app=app) - pp.filterProcessors('copy concat less sass sitemap') - procs = pp.getFilteredProcessors(names) + processors = app.plugin_loader.getProcessors() + procs = get_filtered_processors(processors, names) actual = [p.PROCESSOR_NAME for p in procs] assert sorted(actual) == sorted(expected)
--- a/tests/test_sources_autoconfig.py Thu Jun 25 05:51:47 2015 +1000 +++ b/tests/test_sources_autoconfig.py Sat Jul 11 20:33:55 2015 -0700 @@ -5,49 +5,49 @@ @pytest.mark.parametrize( - 'fs, src_config, expected_paths, expected_metadata', + 'fs_fac, src_config, expected_paths, expected_metadata', [ - (mock_fs(), {}, [], []), - (mock_fs().withPage('test/_index.md'), + (lambda: mock_fs(), {}, [], []), + (lambda: mock_fs().withPage('test/_index.md'), {}, ['_index.md'], [{'slug': '', 'config': {'foo': []}}]), - (mock_fs().withPage('test/something.md'), + (lambda: mock_fs().withPage('test/something.md'), {}, ['something.md'], [{'slug': 'something', 'config': {'foo': []}}]), - (mock_fs().withPage('test/bar/something.md'), + (lambda: mock_fs().withPage('test/bar/something.md'), {}, ['bar/something.md'], [{'slug': 'something', 'config': {'foo': ['bar']}}]), - (mock_fs().withPage('test/bar1/bar2/something.md'), + (lambda: mock_fs().withPage('test/bar1/bar2/something.md'), {}, ['bar1/bar2/something.md'], [{'slug': 'something', 'config': {'foo': ['bar1', 'bar2']}}]), - (mock_fs().withPage('test/something.md'), + (lambda: mock_fs().withPage('test/something.md'), {'collapse_single_values': True}, ['something.md'], [{'slug': 'something', 'config': {'foo': None}}]), - (mock_fs().withPage('test/bar/something.md'), + (lambda: mock_fs().withPage('test/bar/something.md'), {'collapse_single_values': True}, ['bar/something.md'], [{'slug': 'something', 'config': {'foo': 'bar'}}]), - (mock_fs().withPage('test/bar1/bar2/something.md'), + (lambda: mock_fs().withPage('test/bar1/bar2/something.md'), {'collapse_single_values': True}, ['bar1/bar2/something.md'], [{'slug': 'something', 'config': {'foo': ['bar1', 'bar2']}}]), - (mock_fs().withPage('test/something.md'), + (lambda: mock_fs().withPage('test/something.md'), {'only_single_values': True}, ['something.md'], [{'slug': 'something', 'config': {'foo': None}}]), - (mock_fs().withPage('test/bar/something.md'), + (lambda: mock_fs().withPage('test/bar/something.md'), {'only_single_values': True}, ['bar/something.md'], [{'slug': 'something', 'config': {'foo': 'bar'}}]), ]) -def test_autoconfig_source_factories(fs, src_config, expected_paths, +def test_autoconfig_source_factories(fs_fac, src_config, expected_paths, expected_metadata): site_config = { 'sources': { @@ -58,6 +58,7 @@ {'url': '/%slug%', 'source': 'test'}] } site_config['sources']['test'].update(src_config) + fs = fs_fac() fs.withConfig({'site': site_config}) fs.withDir('kitchen/test') with mock_fs_scope(fs): @@ -88,27 +89,27 @@ @pytest.mark.parametrize( - 'fs, expected_paths, expected_metadata', + 'fs_fac, expected_paths, expected_metadata', [ - (mock_fs(), [], []), - (mock_fs().withPage('test/_index.md'), + (lambda: mock_fs(), [], []), + (lambda: mock_fs().withPage('test/_index.md'), ['_index.md'], [{'slug': '', 'config': {'foo': 0, 'foo_trail': [0]}}]), - (mock_fs().withPage('test/something.md'), + (lambda: mock_fs().withPage('test/something.md'), ['something.md'], [{'slug': 'something', 'config': {'foo': 0, 'foo_trail': [0]}}]), - (mock_fs().withPage('test/08_something.md'), + (lambda: mock_fs().withPage('test/08_something.md'), ['08_something.md'], [{'slug': 'something', 'config': {'foo': 8, 'foo_trail': [8]}}]), - (mock_fs().withPage('test/02_there/08_something.md'), + (lambda: mock_fs().withPage('test/02_there/08_something.md'), ['02_there/08_something.md'], [{'slug': 'there/something', 'config': {'foo': 8, 'foo_trail': [2, 8]}}]), ]) -def test_ordered_source_factories(fs, expected_paths, expected_metadata): +def test_ordered_source_factories(fs_fac, expected_paths, expected_metadata): site_config = { 'sources': { 'test': {'type': 'ordered', @@ -117,6 +118,7 @@ 'routes': [ {'url': '/%slug%', 'source': 'test'}] } + fs = fs_fac() fs.withConfig({'site': site_config}) fs.withDir('kitchen/test') with mock_fs_scope(fs): @@ -130,34 +132,34 @@ @pytest.mark.parametrize( - 'fs, route_path, expected_path, expected_metadata', + 'fs_fac, route_path, expected_path, expected_metadata', [ - (mock_fs(), 'missing', None, None), - (mock_fs().withPage('test/something.md'), + (lambda: mock_fs(), 'missing', None, None), + (lambda: mock_fs().withPage('test/something.md'), 'something', 'something.md', {'slug': 'something', 'config': {'foo': 0, 'foo_trail': [0]}}), - (mock_fs().withPage('test/bar/something.md'), + (lambda: mock_fs().withPage('test/bar/something.md'), 'bar/something', 'bar/something.md', {'slug': 'bar/something', 'config': {'foo': 0, 'foo_trail': [0]}}), - (mock_fs().withPage('test/42_something.md'), + (lambda: mock_fs().withPage('test/42_something.md'), 'something', '42_something.md', {'slug': 'something', 'config': {'foo': 42, 'foo_trail': [42]}}), - (mock_fs().withPage('test/bar/42_something.md'), + (lambda: mock_fs().withPage('test/bar/42_something.md'), 'bar/something', 'bar/42_something.md', {'slug': 'bar/something', 'config': {'foo': 42, 'foo_trail': [42]}}), - ((mock_fs() + ((lambda: mock_fs() .withPage('test/42_something.md') .withPage('test/43_other_something.md')), 'something', '42_something.md', {'slug': 'something', 'config': {'foo': 42, 'foo_trail': [42]}}), ]) -def test_ordered_source_find(fs, route_path, expected_path, +def test_ordered_source_find(fs_fac, route_path, expected_path, expected_metadata): site_config = { 'sources': { @@ -167,6 +169,7 @@ 'routes': [ {'url': '/%slug%', 'source': 'test'}] } + fs = fs_fac() fs.withConfig({'site': site_config}) fs.withDir('kitchen/test') with mock_fs_scope(fs):
--- a/tests/test_sources_base.py Thu Jun 25 05:51:47 2015 +1000 +++ b/tests/test_sources_base.py Sat Jul 11 20:33:55 2015 -0700 @@ -6,22 +6,23 @@ from .pathutil import slashfix -@pytest.mark.parametrize('fs, expected_paths, expected_slugs', [ - (mock_fs(), [], []), - (mock_fs().withPage('test/foo.html'), +@pytest.mark.parametrize('fs_fac, expected_paths, expected_slugs', [ + (lambda: mock_fs(), [], []), + (lambda: mock_fs().withPage('test/foo.html'), ['foo.html'], ['foo']), - (mock_fs().withPage('test/foo.md'), + (lambda: mock_fs().withPage('test/foo.md'), ['foo.md'], ['foo']), - (mock_fs().withPage('test/foo.ext'), + (lambda: mock_fs().withPage('test/foo.ext'), ['foo.ext'], ['foo.ext']), - (mock_fs().withPage('test/foo/bar.html'), + (lambda: mock_fs().withPage('test/foo/bar.html'), ['foo/bar.html'], ['foo/bar']), - (mock_fs().withPage('test/foo/bar.md'), + (lambda: mock_fs().withPage('test/foo/bar.md'), ['foo/bar.md'], ['foo/bar']), - (mock_fs().withPage('test/foo/bar.ext'), + (lambda: mock_fs().withPage('test/foo/bar.ext'), ['foo/bar.ext'], ['foo/bar.ext']), ]) -def test_default_source_factories(fs, expected_paths, expected_slugs): +def test_default_source_factories(fs_fac, expected_paths, expected_slugs): + fs = fs_fac() fs.withConfig({ 'site': { 'sources': { @@ -131,7 +132,7 @@ app = fs.getApp() r = PageRef(app, 'whatever:doesnt_exist.md') with pytest.raises(Exception): - r.possible_rel_paths + r.possible_ref_specs def test_page_ref_with_missing_file(): @@ -139,9 +140,11 @@ with mock_fs_scope(fs): app = fs.getApp() r = PageRef(app, 'pages:doesnt_exist.%ext%') - assert r.possible_rel_paths == [ - 'doesnt_exist.html', 'doesnt_exist.md', 'doesnt_exist.textile'] - assert r.source_name == 'pages' + assert r.possible_ref_specs == [ + 'pages:doesnt_exist.html', 'pages:doesnt_exist.md', + 'pages:doesnt_exist.textile'] + with pytest.raises(PageNotFoundError): + r.source_name with pytest.raises(PageNotFoundError): r.rel_path with pytest.raises(PageNotFoundError):
--- a/tests/test_sources_posts.py Thu Jun 25 05:51:47 2015 +1000 +++ b/tests/test_sources_posts.py Sat Jul 11 20:33:55 2015 -0700 @@ -1,44 +1,45 @@ -import os import pytest from .mockutil import mock_fs, mock_fs_scope -@pytest.mark.parametrize('fs, src_type, expected_paths, expected_metadata', [ - (mock_fs(), 'flat', [], []), - (mock_fs().withPage('test/2014-01-01_foo.md'), +@pytest.mark.parametrize('fs_fac, src_type, expected_paths, expected_metadata', [ + (lambda: mock_fs(), 'flat', [], []), + (lambda: mock_fs().withPage('test/2014-01-01_foo.md'), 'flat', ['2014-01-01_foo.md'], [(2014, 1, 1, 'foo')]), - (mock_fs(), 'shallow', [], []), - (mock_fs().withPage('test/2014/01-01_foo.md'), + (lambda: mock_fs(), 'shallow', [], []), + (lambda: mock_fs().withPage('test/2014/01-01_foo.md'), 'shallow', ['2014/01-01_foo.md'], [(2014, 1, 1, 'foo')]), - (mock_fs(), 'hierarchy', [], []), - (mock_fs().withPage('test/2014/01/01_foo.md'), + (lambda: mock_fs(), 'hierarchy', [], []), + (lambda: mock_fs().withPage('test/2014/01/01_foo.md'), 'hierarchy', ['2014/01/01_foo.md'], [(2014, 1, 1, 'foo')]), ]) -def test_post_source_factories(fs, src_type, expected_paths, expected_metadata): - fs.withConfig({ - 'site': { - 'sources': { - 'test': {'type': 'posts/%s' % src_type}}, - 'routes': [ - {'url': '/%slug%', 'source': 'test'}] - } - }) - fs.withDir('kitchen/test') - with mock_fs_scope(fs): - app = fs.getApp(cache=False) - s = app.getSource('test') - facs = list(s.buildPageFactories()) - paths = [f.rel_path for f in facs] - assert paths == expected_paths - metadata = [ - (f.metadata['year'], f.metadata['month'], - f.metadata['day'], f.metadata['slug']) - for f in facs] - assert metadata == expected_metadata +def test_post_source_factories(fs_fac, src_type, expected_paths, + expected_metadata): + fs = fs_fac() + fs.withConfig({ + 'site': { + 'sources': { + 'test': {'type': 'posts/%s' % src_type}}, + 'routes': [ + {'url': '/%slug%', 'source': 'test'}] + } + }) + fs.withDir('kitchen/test') + with mock_fs_scope(fs): + app = fs.getApp(cache=False) + s = app.getSource('test') + facs = list(s.buildPageFactories()) + paths = [f.rel_path for f in facs] + assert paths == expected_paths + metadata = [ + (f.metadata['year'], f.metadata['month'], + f.metadata['day'], f.metadata['slug']) + for f in facs] + assert metadata == expected_metadata
--- a/tests/test_templating_jinjaengine.py Thu Jun 25 05:51:47 2015 +1000 +++ b/tests/test_templating_jinjaengine.py Sat Jul 11 20:33:55 2015 -0700 @@ -37,7 +37,7 @@ def test_layout(): contents = "Blah\n" layout = "{{content}}\nFor site: {{foo}}\n" - expected = "Blah\nFor site: bar" + expected = "Blah\n\nFor site: bar" fs = (mock_fs() .withConfig(app_config) .withAsset('templates/blah.jinja', layout)
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tests/tmpfs.py Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,82 @@ +import os +import os.path +import shutil +import random +from .basefs import TestFileSystemBase + + +class TempDirFileSystem(TestFileSystemBase): + def __init__(self, default_spec=True): + self._root = os.path.join( + os.path.dirname(__file__), + '__tmpfs__', + '%d' % random.randrange(1000)) + self._done = False + if default_spec: + self._initDefaultSpec() + + def path(self, p): + p = p.lstrip('/\\') + return os.path.join(self._root, p) + + def getStructure(self, path=None): + path = self.path(path) + if not os.path.exists(path): + raise Exception("No such path: %s" % path) + if not os.path.isdir(path): + raise Exception("Path is not a directory: %s" % path) + + res = {} + for item in os.listdir(path): + self._getStructureRecursive(res, path, item) + return res + + def getFileEntry(self, path): + path = self.path(path) + with open(path, 'r', encoding='utf8') as fp: + return fp.read() + + def _getStructureRecursive(self, target, parent, cur): + full_cur = os.path.join(parent, cur) + if os.path.isdir(full_cur): + e = {} + for item in os.listdir(full_cur): + self._getStructureRecursive(e, full_cur, item) + target[cur] = e + else: + with open(full_cur, 'r', encoding='utf8') as fp: + target[cur] = fp.read() + + def _createDir(self, path): + if not os.path.exists(path): + os.makedirs(path) + + def _createFile(self, path, contents): + dirpath = os.path.dirname(path) + if not os.path.exists(dirpath): + os.makedirs(dirpath) + with open(path, 'w', encoding='utf8') as fp: + fp.write(contents) + + if not self._done: + import traceback + with open(os.path.join(self._root, 'where.txt'), 'w') as fp: + fp.write('\n'.join(traceback.format_stack(limit=10))) + self._done = True + + +class TempDirScope(object): + def __init__(self, fs, open_patches=None): + self._fs = fs + self._open = open + + @property + def root(self): + return self._fs._root + + def __enter__(self): + return self + + def __exit__(self, type, value, traceback): + shutil.rmtree(self.root) +
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/util/build/generate_messages.cmd Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,11 @@ +@echo off +setlocal + +set CUR_DIR=%~dp0 +set CHEF=%CUR_DIR%..\bin\chef +set OUT_DIR=%CUR_DIR%..\piecrust\resources\messages +set ROOT_DIR=%CUR_DIR%messages + +%CHEF% --root=%ROOT_DIR% bake -o %OUT_DIR% +del %OUT_DIR%\index.html +
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/util/build/generate_messages.sh Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,9 @@ +#!/bin/sh + +CUR_DIR="$( cd "$( dirname "$0" )" && pwd )" +CHEF=${CUR_DIR}/../bin/chef +OUT_DIR=${CUR_DIR}/../piecrust/resources/messages +ROOT_DIR=${CUR_DIR}/messages + +$CHEF --root=$ROOT_DIR bake -o $OUT_DIR +rm ${OUT_DIR}/index.html
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/util/build/messages/config.yml Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,2 @@ +site: + title: PieCrust System Messages
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/util/build/messages/pages/_index.html Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,12 @@ +--- +title: PieCrust System Messages +--- + +Here are the **PieCrust** system message pages: + +* [Requirements Not Met]({{ pcurl('requirements') }}) +* [Error]({{ pcurl('error') }}) +* [Not Found]({{ pcurl('error404') }}) +* [Critical Error]({{ pcurl('critical') }}) + +This very page you're reading, however, is only here for convenience.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/util/build/messages/pages/critical.html Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,6 @@ +--- +title: The Whole Kitchen Burned Down! +layout: error +--- +Something critically bad happened, and **PieCrust** needs to shut down. It's probably our fault. +
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/util/build/messages/pages/error.html Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,5 @@ +--- +title: The Cake Just Burned! +layout: error +--- +
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/util/build/messages/pages/error404.html Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,6 @@ +--- +title: Can't find the sugar! +layout: error +--- +It looks like the page you were trying to access does not exist around here. Try going somewhere else. +
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/util/build/messages/templates/default.html Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,70 @@ +<!doctype html> +<html> +<head> + <title>{{ page.title }}</title> + <meta name="generator" content="PieCrust" /> + <link rel="stylesheet" type="text/css" href="http://fonts.googleapis.com/css?family=Lobster"> + <style> + body { + margin: 0; + padding: 1em; + background: #eee; + color: #000; + font-family: Georgia, serif; + } + h1 { + font-size: 4.5em; + font-family: Lobster, 'Trebuchet MS', Verdana, sans-serif; + text-align: center; + font-weight: bold; + margin-top: 0; + color: #333; + text-shadow: 0px 2px 5px rgba(0,0,0,0.3); + } + h2 { + font-size: 2.5em; + font-family: 'Lobster', 'Trebuchet MS', Verdana, sans-serif; + } + code { + background: #ddd; + padding: 0 0.2em; + } + #preamble { + font-size: 1.2em; + font-style: italic; + text-align: center; + margin-bottom: 0; + } + #container { + margin: 0 20%; + } + #content { + margin: 2em 1em; + } + .error-details { + color: #d11; + } + .note { + margin: 3em; + color: #888; + font-style: italic; + } + </style> +</head> +<body> + <div id="container"> + <div id="header"> + <p id="preamble">A Message From The Kitchen:</p> + <h1>{{ page.title }}</h1> + </div> + <hr /> + <div id="content"> + {% block content %} + {{ content|safe }} + {% endblock %} + </div> + <hr /> + {% block footer %}{% endblock %} + </div> +</body> +</html>
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/util/build/messages/templates/error.html Sat Jul 11 20:33:55 2015 -0700 @@ -0,0 +1,27 @@ +{% extends "default.html" %} + +{% block content %} +{{content|safe}} + +{# The following is `raw` because we want it to be in the + produced page, so it can then be templated on the fly + with the error messages #} +{% raw %} +{% if details %} +<div class="error-details"> + <p>Error details:</p> + <ul> + {% for desc in details %} + <li>{{ desc }}</li> + {% endfor %} + </ul> +</div> +{% endif %} +{% endraw %} +{% endblock %} + +{% block footer %} +{% pcformat 'textile' %} +p(note). You're seeing this because something wrong happend. To see detailed errors with callstacks, run chef with the @--debug@ parameter, append @?!debug@ to the URL, or initialize the @PieCrust@ object with @{'debug': true}@. On the other hand, to see you custom error pages, set the @site/display_errors@ setting to @false@. +{% endpcformat %} +{% endblock %}