piecrust2: piecrust/environment.py annotate

annotate piecrust/environment.py @ 112:d31cbbdb4ecc

Use cache paths that are easier to debug than hashes.

author	Ludovic Chabant <ludovic@chabant.com>
date	Sun, 19 Oct 2014 00:25:02 -0700
parents	208c652551a3
children	371a6c879ab9

rev	line source
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	1 import re
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	2 import time
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	3 import json
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import logging
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	5 import threading
67 563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way. Ludovic Chabant <ludovic@chabant.com> parents: 52 diff changeset	6 import collections
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	7 import repoze.lru
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	9
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	10 logger = logging.getLogger(__name__)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	11
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	13 re_fs_cache_key = re.compile(r'[^\d\w\-\._]+')
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	14
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	15
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	16 def _make_fs_cache_key(key):
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	17 return re_fs_cache_key.sub('_', key)
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	18
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	19
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	20 class MemCache(object):
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	21 """ Simple memory cache. It can be backed by a simple file-system
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	22 cache, but items need to be JSON-serializable to do this.
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	23 """
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	24 def __init__(self, size=2048):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	25 self.cache = repoze.lru.LRUCache(size)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	26 self.lock = threading.RLock()
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	27 self.fs_cache = None
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	28 self._invalidated_fs_items = set()
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	29
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	30 def invalidate(self, key):
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	31 with self.lock:
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	32 logger.debug("Invalidating cache item '%s'." % key)
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	33 self.cache.invalidate(key)
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	34 if self.fs_cache:
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	35 logger.debug("Invalidating FS cache item '%s'." % key)
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	36 fs_key = _make_fs_cache_key(key)
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	37 self._invalidated_fs_items.add(fs_key)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	38
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 47 diff changeset	39 def get(self, key, item_maker, fs_cache_time=None):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	40 item = self.cache.get(key)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	41 if item is None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	42 logger.debug("Acquiring lock for: %s" % key)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	43 with self.lock:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	44 item = self.cache.get(key)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	45 if item is None:
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 47 diff changeset	46 if (self.fs_cache is not None and
fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 47 diff changeset	47 fs_cache_time is not None):
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	48 # Try first from the file-system cache.
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	49 fs_key = _make_fs_cache_key(key)
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	50 if (fs_key not in self._invalidated_fs_items and
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	51 self.fs_cache.isValid(fs_key, fs_cache_time)):
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 47 diff changeset	52 logger.debug("'%s' found in file-system cache." %
fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 47 diff changeset	53 key)
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	54 item_raw = self.fs_cache.read(fs_key)
67 563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way. Ludovic Chabant <ludovic@chabant.com> parents: 52 diff changeset	55 item = json.loads(item_raw,
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way. Ludovic Chabant <ludovic@chabant.com> parents: 52 diff changeset	56 object_pairs_hook=collections.OrderedDict)
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	57 self.cache.put(key, item)
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	58 return item
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	59
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	60 # Look into the mem-cache.
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	61 logger.debug("'%s' not found in cache, must build." % key)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	62 item = item_maker()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	63 self.cache.put(key, item)
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	64
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	65 # Save to the file-system if needed.
50 2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	66 if (self.fs_cache is not None and
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	67 fs_cache_time is not None):
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	68 item_raw = json.dumps(item)
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	69 self.fs_cache.write(fs_key, item_raw)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	70 return item
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	71
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	72
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	73 class ExecutionInfo(object):
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	74 def __init__(self, page, render_ctx):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	75 self.page = page
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	76 self.render_ctx = render_ctx
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	77 self.was_cache_valid = False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	78 self.start_time = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	79
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	80
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	81 class ExecutionInfoStack(threading.local):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	82 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	83 self._page_stack = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	84
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	85 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	86 def current_page_info(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	87 if len(self._page_stack) == 0:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	88 return None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	89 return self._page_stack[-1]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	90
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	91 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	92 def is_main_page(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	93 return len(self._page_stack) == 1
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	94
12 30a42341cfa8 Define page slugs properly, avoid recursions with debug data. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	95 def hasPage(self, page):
30a42341cfa8 Define page slugs properly, avoid recursions with debug data. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	96 for ei in self._page_stack:
30a42341cfa8 Define page slugs properly, avoid recursions with debug data. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	97 if ei.page == page:
30a42341cfa8 Define page slugs properly, avoid recursions with debug data. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	98 return True
30a42341cfa8 Define page slugs properly, avoid recursions with debug data. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	99 return False
30a42341cfa8 Define page slugs properly, avoid recursions with debug data. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	100
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	101 def pushPage(self, page, render_ctx):
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	102 if len(self._page_stack) > 0:
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	103 top = self._page_stack[-1]
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	104 assert top.page is not page
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	105 self._page_stack.append(ExecutionInfo(page, render_ctx))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	106
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	107 def popPage(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	108 del self._page_stack[-1]
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	109
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	110
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	111 class Environment(object):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	112 def __init__(self):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	113 self.start_time = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	114 self.exec_info_stack = ExecutionInfoStack()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	115 self.was_cache_cleaned = False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	116 self.page_repository = MemCache()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	117 self.rendered_segments_repository = MemCache()
111 208c652551a3 Quick fix for making the server correctly update referenced pages. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	118 self.fs_caches = {
208c652551a3 Quick fix for making the server correctly update referenced pages. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	119 'renders': self.rendered_segments_repository}
32 43091c9837bf Fix problems with asset URLs. Ludovic Chabant <ludovic@chabant.com> parents: 12 diff changeset	120 self.base_asset_url_format = '%uri%'
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	121
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	122 def initialize(self, app):
111 208c652551a3 Quick fix for making the server correctly update referenced pages. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	123 for name, repo in self.fs_caches.items():
208c652551a3 Quick fix for making the server correctly update referenced pages. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	124 cache = app.cache.getCache(name)
208c652551a3 Quick fix for making the server correctly update referenced pages. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	125 repo.fs_cache = cache
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	126
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	127
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	128 class StandardEnvironment(Environment):
1 aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	129 def __init__(self):
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	130 super(StandardEnvironment, self).__init__()
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	131

Mercurial > piecrust2

annotate piecrust/environment.py @ 112:d31cbbdb4ecc