piecrust2: piecrust/cache.py annotate

annotate piecrust/cache.py @ 874:f4608e2e80ce

data: Optimize page data creation. `datetime.strftime` was pretty costly so it's no lazily-computed in some places, and replaced with some better alternative elsewhere.

author	Ludovic Chabant <ludovic@chabant.com>
date	Thu, 15 Jun 2017 07:31:50 -0700
parents	4850f8c21b6e
children	d1095774bfcf

rev	line source
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import os
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os.path
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	3 import json
371 c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	4 import shutil
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import codecs
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	6 import hashlib
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	7 import logging
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	8 import repoze.lru
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	9
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	10
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	11 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	12
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	13
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	14 class ExtensibleCache(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	15 def __init__(self, base_dir):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	16 self.base_dir = base_dir
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	17 self.caches = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	18
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	19 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	20 def enabled(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	21 return True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	22
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	23 def getCache(self, name):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	24 c = self.caches.get(name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	25 if c is None:
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	26 c_dir = os.path.join(self.base_dir, name)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	27 if not os.path.isdir(c_dir):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	28 os.makedirs(c_dir, 0o755)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	29
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	30 c = SimpleCache(c_dir)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	31 self.caches[name] = c
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	32 return c
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 5 diff changeset	34 def getCacheDir(self, name):
efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 5 diff changeset	35 return os.path.join(self.base_dir, name)
efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 5 diff changeset	36
105 7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	37 def getCacheNames(self, except_names=None):
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	38 _, dirnames, __ = next(os.walk(self.base_dir))
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	39 if except_names is None:
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	40 return dirnames
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	41 return [dn for dn in dirnames if dn not in except_names]
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	42
371 c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	43 def clearCache(self, name):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	44 cache_dir = self.getCacheDir(name)
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	45 if os.path.isdir(cache_dir):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	46 logger.debug("Cleaning cache: %s" % cache_dir)
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	47 shutil.rmtree(cache_dir)
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	48
424 5feb71d31a4f internal: Fix caches being orphaned from their directory. Ludovic Chabant <ludovic@chabant.com> parents: 416 diff changeset	49 # Re-create the cache-dir because now our Cache instance points
5feb71d31a4f internal: Fix caches being orphaned from their directory. Ludovic Chabant <ludovic@chabant.com> parents: 416 diff changeset	50 # to a directory that doesn't exist anymore.
5feb71d31a4f internal: Fix caches being orphaned from their directory. Ludovic Chabant <ludovic@chabant.com> parents: 416 diff changeset	51 os.makedirs(cache_dir, 0o755)
5feb71d31a4f internal: Fix caches being orphaned from their directory. Ludovic Chabant <ludovic@chabant.com> parents: 416 diff changeset	52
371 c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	53 def clearCaches(self, except_names=None):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	54 for name in self.getCacheNames(except_names=except_names):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	55 self.clearCache(name)
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	56
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	57
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58 class SimpleCache(object):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 def __init__(self, base_dir):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	60 self.base_dir = base_dir
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	61 if not os.path.isdir(base_dir):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	62 raise Exception("Cache directory doesn't exist: %s" % base_dir)
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64 def isValid(self, path, time):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	65 cache_time = self.getCacheTime(path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	66 if cache_time is None:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	67 return False
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 if isinstance(time, list):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69 for t in time:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	70 if cache_time < t:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71 return False
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	72 return True
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	73 return cache_time >= time
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	74
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75 def getCacheTime(self, path):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76 cache_path = self.getCachePath(path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	77 try:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	78 return os.path.getmtime(cache_path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	79 except os.error:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80 return None
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	81
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	82 def has(self, path):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	83 cache_path = self.getCachePath(path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84 return os.path.isfile(cache_path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	85
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	86 def read(self, path):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	87 cache_path = self.getCachePath(path)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	88 logger.debug("Reading cache: %s" % cache_path)
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	89 with codecs.open(cache_path, 'r', 'utf-8') as fp:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	90 return fp.read()
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	91
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	92 def write(self, path, content):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	93 cache_path = self.getCachePath(path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	94 cache_dir = os.path.dirname(cache_path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	95 if not os.path.isdir(cache_dir):
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	96 os.makedirs(cache_dir, 0o755)
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 5 diff changeset	97 logger.debug("Writing cache: %s" % cache_path)
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	98 with codecs.open(cache_path, 'w', 'utf-8') as fp:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	99 fp.write(content)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	100
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	101 def getCachePath(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	102 if path.startswith('.'):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	103 path = '__index__' + path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	104 return os.path.join(self.base_dir, path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	105
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	106
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	107 class NullCache(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	108 def isValid(self, path, time):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	109 return False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	110
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	111 def getCacheTime(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	112 return None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	113
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	114 def has(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	115 return False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	116
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	117 def read(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	118 raise Exception("Null cache has no data.")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	119
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	120 def write(self, path, content):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	121 pass
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	122
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	123 def getCachePath(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	124 raise Exception("Null cache can't make paths.")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	125
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	126
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	127 class NullExtensibleCache(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	128 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	129 self.null_cache = NullCache()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	130
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	131 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	132 def enabled(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	133 return False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	134
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	135 def getCache(self, name):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	136 return self.null_cache
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	137
371 c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	138 def getCacheDir(self, name):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	139 raise NotImplementedError()
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	140
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	141 def getCacheNames(self, except_names=None):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	142 return []
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	143
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	144 def clearCache(self, name):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	145 pass
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	146
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	147 def clearCaches(self, except_names=None):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	148 pass
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	149
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	150
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	151 def _make_fs_cache_key(key):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	152 return hashlib.md5(key.encode('utf8')).hexdigest()
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	153
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	154
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	155 class MemCache(object):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	156 """ Simple memory cache. It can be backed by a simple file-system
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	157 cache, but items need to be JSON-serializable to do this.
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	158 """
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	159 def __init__(self, size=2048):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	160 self.cache = repoze.lru.LRUCache(size)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	161 self.fs_cache = None
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	162 self._last_access_hit = None
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	163 self._invalidated_fs_items = set()
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	164 self._missed_keys = []
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	165 self._misses = 0
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	166 self._hits = 0
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	167
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	168 @property
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	169 def last_access_hit(self):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	170 return self._last_access_hit
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	171
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	172 def invalidate(self, key):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	173 logger.debug("Invalidating cache item '%s'." % key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	174 self.cache.invalidate(key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	175 if self.fs_cache:
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	176 logger.debug("Invalidating FS cache item '%s'." % key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	177 fs_key = _make_fs_cache_key(key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	178 self._invalidated_fs_items.add(fs_key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	179
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	180 def put(self, key, item, save_to_fs=True):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	181 self.cache.put(key, item)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	182 if self.fs_cache and save_to_fs:
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	183 fs_key = _make_fs_cache_key(key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	184 item_raw = json.dumps(item)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	185 self.fs_cache.write(fs_key, item_raw)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	186
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	187 def get(self, key, item_maker, fs_cache_time=None, save_to_fs=True):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	188 self._last_access_hit = True
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	189 item = self.cache.get(key)
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	190 if item is not None:
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	191 self._hits += 1
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	192 return item
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	193
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	194 if (self.fs_cache is not None and
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	195 fs_cache_time is not None):
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	196 # Try first from the file-system cache.
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	197 fs_key = _make_fs_cache_key(key)
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	198 if (fs_key not in self._invalidated_fs_items and
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	199 self.fs_cache.isValid(fs_key, fs_cache_time)):
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	200 logger.debug("'%s' found in file-system cache." %
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	201 key)
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	202 item_raw = self.fs_cache.read(fs_key)
719 a066f4ac9094 rendering: Use `fastpickle` serialization before JSON. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	203 item = json.loads(item_raw)
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	204 self.cache.put(key, item)
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	205 self._hits += 1
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	206 return item
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	207
687 61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	208 # Look into the mem-cache.
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	209 logger.debug("'%s' not found in cache, must build." % key)
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	210 item = item_maker()
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	211 self.cache.put(key, item)
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	212 self._last_access_hit = False
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	213 self._misses += 1
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	214 self._missed_keys.append(key)
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	215
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	216 # Save to the file-system if needed.
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	217 if self.fs_cache is not None and save_to_fs:
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	218 item_raw = json.dumps(item)
61d606fbc313 bake: Change `show-timers` to `show-stats`, add stats. Ludovic Chabant <ludovic@chabant.com> parents: 424 diff changeset	219 self.fs_cache.write(fs_key, item_raw)
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	220
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	221 return item
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	222
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	223

Mercurial > piecrust2

annotate piecrust/cache.py @ 874:f4608e2e80ce