piecrust2: piecrust/environment.py annotate

annotate piecrust/environment.py @ 111:208c652551a3

Quick fix for making the server correctly update referenced pages. Disable the file-system cache for rendered segments when in server mode. We can bring this optimization back when we're actually using the baking record in the server too in order to know dependencies.

author	Ludovic Chabant <ludovic@chabant.com>
date	Thu, 16 Oct 2014 17:03:42 -0700
parents	0445a2232de7
children	371a6c879ab9

rev	line source
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	1 import re
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	2 import time
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	3 import json
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import logging
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	5 import threading
67 563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way. Ludovic Chabant <ludovic@chabant.com> parents: 52 diff changeset	6 import collections
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	7 import repoze.lru
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	9
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	10 logger = logging.getLogger(__name__)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	11
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	13 re_fs_cache_key = re.compile(r'[^\d\w\-\._]+')
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	14
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	15
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	16 def _make_fs_cache_key(key):
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	17 return re_fs_cache_key.sub('_', key)
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	18
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	19
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	20 class MemCache(object):
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	21 """ Simple memory cache. It can be backed by a simple file-system
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	22 cache, but items need to be JSON-serializable to do this.
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	23 """
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	24 def __init__(self, size=2048):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	25 self.cache = repoze.lru.LRUCache(size)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	26 self.lock = threading.RLock()
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	27 self.fs_cache = None
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	28 self._invalidated_fs_items = set()
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	29
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	30 def invalidate(self, key):
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	31 with self.lock:
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	32 logger.debug("Invalidating cache item '%s'." % key)
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	33 self.cache.invalidate(key)
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	34 if self.fs_cache:
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	35 logger.debug("Invalidating FS cache item '%s'." % key)
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	36 fs_key = _make_fs_cache_key(key)
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	37 self._invalidated_fs_items.add(fs_key)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	38
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 47 diff changeset	39 def get(self, key, item_maker, fs_cache_time=None):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	40 item = self.cache.get(key)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	41 if item is None:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	42 logger.debug("Acquiring lock for: %s" % key)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	43 with self.lock:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	44 item = self.cache.get(key)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	45 if item is None:
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 47 diff changeset	46 if (self.fs_cache is not None and
fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 47 diff changeset	47 fs_cache_time is not None):
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	48 # Try first from the file-system cache.
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	49 fs_key = _make_fs_cache_key(key)
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	50 if (fs_key not in self._invalidated_fs_items and
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	51 self.fs_cache.isValid(fs_key, fs_cache_time)):
49 fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 47 diff changeset	52 logger.debug("'%s' found in file-system cache." %
fce061f8c2ed Fix cache validation issue with rendered segments, limit disk access. Ludovic Chabant <ludovic@chabant.com> parents: 47 diff changeset	53 key)
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	54 item_raw = self.fs_cache.read(fs_key)
67 563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way. Ludovic Chabant <ludovic@chabant.com> parents: 52 diff changeset	55 item = json.loads(item_raw,
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way. Ludovic Chabant <ludovic@chabant.com> parents: 52 diff changeset	56 object_pairs_hook=collections.OrderedDict)
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	57 self.cache.put(key, item)
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	58 return item
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	59
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	60 # Look into the mem-cache.
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	61 logger.debug("'%s' not found in cache, must build." % key)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	62 item = item_maker()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	63 self.cache.put(key, item)
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	64
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	65 # Save to the file-system if needed.
50 2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	66 if (self.fs_cache is not None and
2fec3ee1298f Properly override pages between realms. Ludovic Chabant <ludovic@chabant.com> parents: 49 diff changeset	67 fs_cache_time is not None):
47 558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	68 item_raw = json.dumps(item)
558e3602be40 Cache rendered segments to disk. Ludovic Chabant <ludovic@chabant.com> parents: 32 diff changeset	69 self.fs_cache.write(fs_key, item_raw)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	70 return item
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	71
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	72
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	73 class ExecutionInfo(object):
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	74 def __init__(self, page, render_ctx):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	75 self.page = page
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	76 self.render_ctx = render_ctx
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	77 self.was_cache_valid = False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	78 self.start_time = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	79
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	80
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	81 class ExecutionInfoStack(threading.local):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	82 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	83 self._page_stack = []
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	84
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	85 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	86 def current_page_info(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	87 if len(self._page_stack) == 0:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	88 return None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	89 return self._page_stack[-1]
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	90
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	91 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	92 def is_main_page(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	93 return len(self._page_stack) == 1
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	94
12 30a42341cfa8 Define page slugs properly, avoid recursions with debug data. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	95 def hasPage(self, page):
30a42341cfa8 Define page slugs properly, avoid recursions with debug data. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	96 for ei in self._page_stack:
30a42341cfa8 Define page slugs properly, avoid recursions with debug data. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	97 if ei.page == page:
30a42341cfa8 Define page slugs properly, avoid recursions with debug data. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	98 return True
30a42341cfa8 Define page slugs properly, avoid recursions with debug data. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	99 return False
30a42341cfa8 Define page slugs properly, avoid recursions with debug data. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	100
96 0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	101 def pushPage(self, page, render_ctx):
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	102 if len(self._page_stack) > 0:
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	103 top = self._page_stack[-1]
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	104 assert top.page is not page
0445a2232de7 Improvements and fixes to incremental baking. Ludovic Chabant <ludovic@chabant.com> parents: 67 diff changeset	105 self._page_stack.append(ExecutionInfo(page, render_ctx))
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	106
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	107 def popPage(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	108 del self._page_stack[-1]
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	109
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	110
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	111 class Environment(object):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	112 def __init__(self):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	113 self.start_time = time.clock()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	114 self.exec_info_stack = ExecutionInfoStack()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	115 self.was_cache_cleaned = False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	116 self.page_repository = MemCache()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 1 diff changeset	117 self.rendered_segments_repository = MemCache()
111 208c652551a3 Quick fix for making the server correctly update referenced pages. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	118 self.fs_caches = {
208c652551a3 Quick fix for making the server correctly update referenced pages. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	119 'renders': self.rendered_segments_repository}
32 43091c9837bf Fix problems with asset URLs. Ludovic Chabant <ludovic@chabant.com> parents: 12 diff changeset	120 self.base_asset_url_format = '%uri%'
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	121
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	122 def initialize(self, app):
111 208c652551a3 Quick fix for making the server correctly update referenced pages. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	123 for name, repo in self.fs_caches.items():
208c652551a3 Quick fix for making the server correctly update referenced pages. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	124 cache = app.cache.getCache(name)
208c652551a3 Quick fix for making the server correctly update referenced pages. Ludovic Chabant <ludovic@chabant.com> parents: 96 diff changeset	125 repo.fs_cache = cache
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	126
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	127
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	128 class StandardEnvironment(Environment):
1 aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	129 def __init__(self):
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	130 super(StandardEnvironment, self).__init__()
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	131

Mercurial > piecrust2

annotate piecrust/environment.py @ 111:208c652551a3