piecrust2: piecrust/page.py annotate

annotate piecrust/page.py @ 411:e7b865f8f335

bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.

author	Ludovic Chabant <ludovic@chabant.com>
date	Fri, 12 Jun 2015 17:09:19 -0700
parents	dd25bd3ce1f9
children	32c7c2d219d2

rev	line source
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import re
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import sys
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import json
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import codecs
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import os.path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 import hashlib
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	7 import logging
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8 import datetime
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	9 import dateutil.parser
67 563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	10 import collections
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 7 diff changeset	11 from werkzeug.utils import cached_property
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	12 from piecrust.configuration import (
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	13 Configuration, ConfigurationError,
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	14 parse_config_header)
147 ab6e7e0e9d44 Pass date information to routing when building URLs. Ludovic Chabant <ludovic@chabant.com> parents: 142 diff changeset	15 from piecrust.routing import IRouteMetadataProvider
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21 class PageConfiguration(Configuration):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22 def __init__(self, values=None, validate=True):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	23 super(PageConfiguration, self).__init__(values, validate)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	25 def _validateAll(self, values):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26 values.setdefault('title', 'Untitled Page')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27 values.setdefault('content_type', 'html')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 ppp = values.get('posts_per_page')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29 if ppp is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	30 values.setdefault('items_per_page', ppp)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31 pf = values.get('posts_filters')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32 if pf is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 values.setdefault('items_filters', pf)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34 return values
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	37 FLAG_NONE = 0
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	38 FLAG_RAW_CACHE_VALID = 2**0
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	39
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	40
147 ab6e7e0e9d44 Pass date information to routing when building URLs. Ludovic Chabant <ludovic@chabant.com> parents: 142 diff changeset	41 class Page(IRouteMetadataProvider):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42 def __init__(self, source, source_metadata, rel_path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	43 self.source = source
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44 self.source_metadata = source_metadata
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	45 self.rel_path = rel_path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	46 self._config = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	47 self._raw_content = None
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	48 self._flags = FLAG_NONE
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 self._datetime = None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52 def app(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53 return self.source.app
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	55 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	56 def ref_spec(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	57 return '%s:%s' % (self.source.name, self.rel_path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 7 diff changeset	59 @cached_property
fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 7 diff changeset	60 def path(self):
363 dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 336 diff changeset	61 path, _ = self.source.resolveRef(self.rel_path)
dd25bd3ce1f9 serve: Refactoring and fixes to be able to serve taxonomy pages. Ludovic Chabant <ludovic@chabant.com> parents: 336 diff changeset	62 return path
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 7 diff changeset	63
fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 7 diff changeset	64 @cached_property
fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 7 diff changeset	65 def path_mtime(self):
fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 7 diff changeset	66 return os.path.getmtime(self.path)
fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 7 diff changeset	67
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 @property
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	69 def flags(self):
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	70 return self._flags
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	71
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	72 @property
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	73 def config(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	74 self._load()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75 return self._config
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	77 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	78 def raw_content(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	79 self._load()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80 return self._raw_content
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	81
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	82 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	83 def datetime(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84 if self._datetime is None:
142 0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	85 try:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	86 if 'datetime' in self.source_metadata:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	87 # Get the date/time from the source.
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	88 self._datetime = self.source_metadata['datetime']
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	89 elif 'date' in self.source_metadata:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	90 # Get the date from the source. Potentially get the
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	91 # time from the page config.
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	92 page_date = self.source_metadata['date']
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	93 page_time = _parse_config_time(self.config.get('time'))
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	94 if page_time is not None:
108 b6ec402d32bb Properly add the config time to a page's datetime. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	95 self._datetime = datetime.datetime(
b6ec402d32bb Properly add the config time to a page's datetime. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	96 page_date.year,
b6ec402d32bb Properly add the config time to a page's datetime. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	97 page_date.month,
b6ec402d32bb Properly add the config time to a page's datetime. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	98 page_date.day) + page_time
142 0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	99 else:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	100 self._datetime = datetime.datetime(
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	101 page_date.year, page_date.month, page_date.day)
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	102 elif 'date' in self.config:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	103 # Get the date from the page config, and maybe the
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	104 # time too.
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	105 page_date = _parse_config_date(self.config.get('date'))
90 e293f08d954e PyYAML supports sexagesimal notation, so handle that for page times. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	106 self._datetime = datetime.datetime(
142 0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	107 page_date.year,
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	108 page_date.month,
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	109 page_date.day)
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	110 page_time = _parse_config_time(self.config.get('time'))
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	111 if page_time is not None:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	112 self._datetime += page_time
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	113 else:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	114 # No idea what the date/time for this page is.
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	115 self._datetime = datetime.datetime.fromtimestamp(0)
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	116 except Exception as ex:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	117 raise Exception(
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	118 "Error computing time for page: %s" %
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	119 self.path) from ex
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	120 return self._datetime
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	121
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	122 @datetime.setter
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	123 def datetime(self, value):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	124 self._datetime = value
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	125
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	126 def getSegment(self, name='content'):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	127 return self.raw_content[name]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	128
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	129 def _load(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	130 if self._config is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	131 return
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	132
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	133 config, content, was_cache_valid = load_page(self.app, self.path,
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	134 self.path_mtime)
142 0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	135 if 'config' in self.source_metadata:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	136 config.merge(self.source_metadata['config'])
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	137
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	138 self._config = config
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	139 self._raw_content = content
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	140 if was_cache_valid:
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	141 self._flags \|= FLAG_RAW_CACHE_VALID
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	142
147 ab6e7e0e9d44 Pass date information to routing when building URLs. Ludovic Chabant <ludovic@chabant.com> parents: 142 diff changeset	143 def getRouteMetadata(self):
ab6e7e0e9d44 Pass date information to routing when building URLs. Ludovic Chabant <ludovic@chabant.com> parents: 142 diff changeset	144 page_dt = self.datetime
ab6e7e0e9d44 Pass date information to routing when building URLs. Ludovic Chabant <ludovic@chabant.com> parents: 142 diff changeset	145 return {
ab6e7e0e9d44 Pass date information to routing when building URLs. Ludovic Chabant <ludovic@chabant.com> parents: 142 diff changeset	146 'year': page_dt.year,
ab6e7e0e9d44 Pass date information to routing when building URLs. Ludovic Chabant <ludovic@chabant.com> parents: 142 diff changeset	147 'month': page_dt.month,
ab6e7e0e9d44 Pass date information to routing when building URLs. Ludovic Chabant <ludovic@chabant.com> parents: 142 diff changeset	148 'day': page_dt.day}
ab6e7e0e9d44 Pass date information to routing when building URLs. Ludovic Chabant <ludovic@chabant.com> parents: 142 diff changeset	149
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150
142 0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	151 def _parse_config_date(page_date):
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	152 if page_date is None:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	153 return None
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	154
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	155 if isinstance(page_date, str):
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	156 try:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	157 parsed_d = dateutil.parser.parse(page_date)
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	158 except Exception as ex:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	159 raise ConfigurationError("Invalid date: %s" % page_date) from ex
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	160 return datetime.date(
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	161 year=parsed_d.year,
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	162 month=parsed_d.month,
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	163 day=parsed_d.day)
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	164
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	165 raise ConfigurationError("Invalid date: %s" % page_date)
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	166
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	167
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	168 def _parse_config_time(page_time):
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	169 if page_time is None:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	170 return None
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	171
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	172 if isinstance(page_time, datetime.timedelta):
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	173 return page_time
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	174
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	175 if isinstance(page_time, str):
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	176 try:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	177 parsed_t = dateutil.parser.parse(page_time)
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	178 except Exception as ex:
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	179 raise ConfigurationError("Invalid time: %s" % page_time) from ex
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	180 return datetime.timedelta(
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	181 hours=parsed_t.hour,
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	182 minutes=parsed_t.minute,
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	183 seconds=parsed_t.second)
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	184
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	185 if isinstance(page_time, int):
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	186 # Total seconds... convert to a time struct.
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	187 return datetime.timedelta(seconds=page_time)
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	188
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	189 raise ConfigurationError("Invalid time: %s" % page_time)
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	190
0bdd938d6b9f Better date/time handling for pages: Ludovic Chabant <ludovic@chabant.com> parents: 128 diff changeset	191
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	192 class PageLoadingError(Exception):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	193 def __init__(self, path, inner=None):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	194 super(PageLoadingError, self).__init__(
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	195 "Error loading page: %s" % path,
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	196 inner)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	197
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	198
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	199 class ContentSegment(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	200 debug_render_func = 'debug_render'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	201
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	202 def __init__(self, content=None, fmt=None):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	203 self.parts = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	204 if content is not None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	205 self.parts.append(ContentSegmentPart(content, fmt))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	206
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	207 def debug_render(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	208 return '\n'.join([p.content for p in self.parts])
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	209
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	210
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	211 class ContentSegmentPart(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	212 def __init__(self, content, fmt=None, line=-1):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	213 self.content = content
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	214 self.fmt = fmt
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	215 self.line = line
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	216
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	217 def __str__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	218 return '%s [%s]' % (self.content, self.fmt or '<default>')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	219
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	220
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	221 def json_load_segments(data):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	222 segments = {}
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	223 for key, seg_data in data.items():
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	224 seg = ContentSegment()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	225 for p_data in seg_data:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	226 part = ContentSegmentPart(p_data['c'], p_data['f'], p_data['l'])
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	227 seg.parts.append(part)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	228 segments[key] = seg
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	229 return segments
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	230
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	231
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	232 def json_save_segments(segments):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	233 data = {}
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	234 for key, seg in segments.items():
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	235 seg_data = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	236 for part in seg.parts:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	237 p_data = {'c': part.content, 'f': part.fmt, 'l': part.line}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	238 seg_data.append(p_data)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	239 data[key] = seg_data
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	240 return data
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	241
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	242
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 7 diff changeset	243 def load_page(app, path, path_mtime=None):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	244 try:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	245 with app.env.timerScope('PageLoad'):
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	246 return _do_load_page(app, path, path_mtime)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	247 except Exception as e:
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	248 logger.exception(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	249 "Error loading page: %s" %
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	250 os.path.relpath(path, app.root_dir))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	251 _, __, traceback = sys.exc_info()
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	252 raise PageLoadingError(path, e).with_traceback(traceback)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	253
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	254
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 7 diff changeset	255 def _do_load_page(app, path, path_mtime):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	256 # Check the cache first.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	257 cache = app.cache.getCache('pages')
336 aa6b7ff6a193 internal: Use hashes for cache paths. Ludovic Chabant <ludovic@chabant.com> parents: 147 diff changeset	258 cache_path = hashlib.md5(path.encode('utf8')).hexdigest() + '.json'
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 7 diff changeset	259 page_time = path_mtime or os.path.getmtime(path)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	260 if cache.isValid(cache_path, page_time):
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	261 cache_data = json.loads(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	262 cache.read(cache_path),
67 563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	263 object_pairs_hook=collections.OrderedDict)
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	264 config = PageConfiguration(
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	265 values=cache_data['config'],
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	266 validate=False)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	267 content = json_load_segments(cache_data['content'])
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	268 return config, content, True
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	269
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	270 # Nope, load the page from the source file.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	271 logger.debug("Loading page configuration from: %s" % path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	272 with codecs.open(path, 'r', 'utf-8') as fp:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	273 raw = fp.read()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	274 header, offset = parse_config_header(raw)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	275
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 363 diff changeset	276 if 'format' not in header:
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	277 auto_formats = app.config.get('site/auto_formats')
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	278 name, ext = os.path.splitext(path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	279 header['format'] = auto_formats.get(ext, None)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	280
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	281 config = PageConfiguration(header)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	282 content = parse_segments(raw, offset)
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	283 config.set('segments', list(content.keys()))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	284
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	285 # Save to the cache.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	286 cache_data = {
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	287 'config': config.get(),
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	288 'content': json_save_segments(content)}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	289 cache.write(cache_path, json.dumps(cache_data))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	290
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 90 diff changeset	291 return config, content, False
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	292
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	293
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	294 segment_pattern = re.compile(
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	295 r"""^\-\-\-\s(?P<name>\w+)(\:(?P<fmt>\w+))?\s\-\-\-\s*$""",
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	296 re.M)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	297 part_pattern = re.compile(
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	298 r"""^<\-\-\s(?P<fmt>\w+)\s\-\->\s*$""",
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	299 re.M)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	300
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	301
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	302 def _count_lines(s):
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	303 return len(s.split('\n'))
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	304
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	305
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	306 def parse_segments(raw, offset=0):
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	307 # Get the number of lines in the header.
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	308 header_lines = _count_lines(raw[:offset].rstrip())
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	309 current_line = header_lines
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	310
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	311 # Start parsing.
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	312 matches = list(segment_pattern.finditer(raw, offset))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	313 num_matches = len(matches)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	314 if num_matches > 0:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	315 contents = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	316
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	317 first_offset = matches[0].start()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	318 if first_offset > 0:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	319 # There's some default content segment at the beginning.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	320 seg = ContentSegment()
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	321 seg.parts, current_line = parse_segment_parts(
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	322 raw, offset, first_offset, current_line)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	323 contents['content'] = seg
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	324
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	325 for i in range(1, num_matches):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	326 m1 = matches[i - 1]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	327 m2 = matches[i]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	328 seg = ContentSegment()
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	329 seg.parts, current_line = parse_segment_parts(
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	330 raw, m1.end() + 1, m2.start(), current_line,
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	331 m1.group('fmt'))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	332 contents[m1.group('name')] = seg
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	333
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	334 # Handle text past the last match.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	335 lastm = matches[-1]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	336 seg = ContentSegment()
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	337 seg.parts, current_line = parse_segment_parts(
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	338 raw, lastm.end() + 1, len(raw), current_line,
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	339 lastm.group('fmt'))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	340 contents[lastm.group('name')] = seg
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	341
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	342 return contents
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	343 else:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	344 # No segments, just content.
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	345 seg = ContentSegment()
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	346 seg.parts, current_line = parse_segment_parts(
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	347 raw, offset, len(raw), current_line)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	348 return {'content': seg}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	349
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	350
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	351 def parse_segment_parts(raw, start, end, line_offset, first_part_fmt=None):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	352 matches = list(part_pattern.finditer(raw, start, end))
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	353 num_matches = len(matches)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	354 if num_matches > 0:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	355 parts = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	356
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	357 # First part, before the first format change.
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	358 part_text = raw[start:matches[0].start()]
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	359 parts.append(
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	360 ContentSegmentPart(part_text, first_part_fmt, line_offset))
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	361 line_offset += _count_lines(part_text)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	362
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	363 for i in range(1, num_matches):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	364 m1 = matches[i - 1]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	365 m2 = matches[i]
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	366 part_text = raw[m1.end() + 1:m2.start()]
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	367 parts.append(
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	368 ContentSegmentPart(
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	369 part_text, m1.group('fmt'), line_offset))
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	370 line_offset += _count_lines(part_text)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	371
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	372 lastm = matches[-1]
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	373 part_text = raw[lastm.end() + 1:end]
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	374 parts.append(ContentSegmentPart(
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	375 part_text, lastm.group('fmt'), line_offset))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	376
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	377 return parts, line_offset
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	378 else:
128 28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	379 part_text = raw[start:end]
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	380 parts = [ContentSegmentPart(part_text, first_part_fmt, line_offset)]
28444014ce7d Fix error reporting and counting of lines. Ludovic Chabant <ludovic@chabant.com> parents: 112 diff changeset	381 return parts, line_offset
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	382

Mercurial > piecrust2

annotate piecrust/page.py @ 411:e7b865f8f335