piecrust2: piecrust/cache.py annotate

annotate piecrust/cache.py @ 550:6f216c1ab6b1

bake: Add a flag to know which record entries got collapsed from last run. This makes it possible to find entries for things that were actually baked during the current run, as opposed to skipped because they were "clean".

author	Ludovic Chabant <ludovic@chabant.com>
date	Tue, 04 Aug 2015 21:22:30 -0700
parents	5feb71d31a4f
children	61d606fbc313

rev	line source
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import os
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os.path
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	3 import json
371 c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	4 import shutil
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import codecs
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	6 import hashlib
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	7 import logging
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	8 import collections
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	9 import repoze.lru
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	10
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	11
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	12 logger = logging.getLogger(__name__)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	13
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	14
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	15 class ExtensibleCache(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	16 def __init__(self, base_dir):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	17 self.base_dir = base_dir
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	18 self.caches = {}
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	20 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	21 def enabled(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	22 return True
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	23
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	24 def getCache(self, name):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	25 c = self.caches.get(name)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	26 if c is None:
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	27 c_dir = os.path.join(self.base_dir, name)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	28 if not os.path.isdir(c_dir):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	29 os.makedirs(c_dir, 0o755)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	30
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	31 c = SimpleCache(c_dir)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	32 self.caches[name] = c
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	33 return c
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 5 diff changeset	35 def getCacheDir(self, name):
efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 5 diff changeset	36 return os.path.join(self.base_dir, name)
efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 5 diff changeset	37
105 7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	38 def getCacheNames(self, except_names=None):
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	39 _, dirnames, __ = next(os.walk(self.base_dir))
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	40 if except_names is None:
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	41 return dirnames
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	42 return [dn for dn in dirnames if dn not in except_names]
7d2fdf43d7ca Property clean all caches when force baking, except the `app` cache. Ludovic Chabant <ludovic@chabant.com> parents: 45 diff changeset	43
371 c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	44 def clearCache(self, name):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	45 cache_dir = self.getCacheDir(name)
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	46 if os.path.isdir(cache_dir):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	47 logger.debug("Cleaning cache: %s" % cache_dir)
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	48 shutil.rmtree(cache_dir)
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	49
424 5feb71d31a4f internal: Fix caches being orphaned from their directory. Ludovic Chabant <ludovic@chabant.com> parents: 416 diff changeset	50 # Re-create the cache-dir because now our Cache instance points
5feb71d31a4f internal: Fix caches being orphaned from their directory. Ludovic Chabant <ludovic@chabant.com> parents: 416 diff changeset	51 # to a directory that doesn't exist anymore.
5feb71d31a4f internal: Fix caches being orphaned from their directory. Ludovic Chabant <ludovic@chabant.com> parents: 416 diff changeset	52 os.makedirs(cache_dir, 0o755)
5feb71d31a4f internal: Fix caches being orphaned from their directory. Ludovic Chabant <ludovic@chabant.com> parents: 416 diff changeset	53
371 c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	54 def clearCaches(self, except_names=None):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	55 for name in self.getCacheNames(except_names=except_names):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	56 self.clearCache(name)
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	57
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 class SimpleCache(object):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	60 def __init__(self, base_dir):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	61 self.base_dir = base_dir
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62 if not os.path.isdir(base_dir):
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	63 raise Exception("Cache directory doesn't exist: %s" % base_dir)
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	65 def isValid(self, path, time):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	66 cache_time = self.getCacheTime(path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	67 if cache_time is None:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 return False
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69 if isinstance(time, list):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	70 for t in time:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71 if cache_time < t:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	72 return False
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	73 return True
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	74 return cache_time >= time
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76 def getCacheTime(self, path):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	77 cache_path = self.getCachePath(path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	78 try:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	79 return os.path.getmtime(cache_path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80 except os.error:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	81 return None
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	82
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	83 def has(self, path):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84 cache_path = self.getCachePath(path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	85 return os.path.isfile(cache_path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	86
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	87 def read(self, path):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	88 cache_path = self.getCachePath(path)
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	89 logger.debug("Reading cache: %s" % cache_path)
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	90 with codecs.open(cache_path, 'r', 'utf-8') as fp:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	91 return fp.read()
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	92
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	93 def write(self, path, content):
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	94 cache_path = self.getCachePath(path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	95 cache_dir = os.path.dirname(cache_path)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	96 if not os.path.isdir(cache_dir):
5 474c9882decf Upgrade to Python 3. Ludovic Chabant <ludovic@chabant.com> parents: 3 diff changeset	97 os.makedirs(cache_dir, 0o755)
45 efd0d3bacc9e Don't recursively clean the cache. Ludovic Chabant <ludovic@chabant.com> parents: 5 diff changeset	98 logger.debug("Writing cache: %s" % cache_path)
0 a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	99 with codecs.open(cache_path, 'w', 'utf-8') as fp:
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	100 fp.write(content)
a212a3f2e3ee Initial commit. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	101
3 f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	102 def getCachePath(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	103 if path.startswith('.'):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	104 path = '__index__' + path
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	105 return os.path.join(self.base_dir, path)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	106
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	107
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	108 class NullCache(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	109 def isValid(self, path, time):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	110 return False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	111
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	112 def getCacheTime(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	113 return None
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	114
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	115 def has(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	116 return False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	117
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	118 def read(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	119 raise Exception("Null cache has no data.")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	120
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	121 def write(self, path, content):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	122 pass
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	123
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	124 def getCachePath(self, path):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	125 raise Exception("Null cache can't make paths.")
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	126
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	127
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	128 class NullExtensibleCache(object):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	129 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	130 self.null_cache = NullCache()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	131
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	132 @property
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	133 def enabled(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	134 return False
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	135
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	136 def getCache(self, name):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	137 return self.null_cache
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional. Ludovic Chabant <ludovic@chabant.com> parents: 0 diff changeset	138
371 c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	139 def getCacheDir(self, name):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	140 raise NotImplementedError()
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	141
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	142 def getCacheNames(self, except_names=None):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	143 return []
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	144
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	145 def clearCache(self, name):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	146 pass
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	147
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	148 def clearCaches(self, except_names=None):
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	149 pass
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts. Ludovic Chabant <ludovic@chabant.com> parents: 105 diff changeset	150
416 ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	151
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	152 def _make_fs_cache_key(key):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	153 return hashlib.md5(key.encode('utf8')).hexdigest()
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	154
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	155
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	156 class MemCache(object):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	157 """ Simple memory cache. It can be backed by a simple file-system
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	158 cache, but items need to be JSON-serializable to do this.
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	159 """
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	160 def __init__(self, size=2048):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	161 self.cache = repoze.lru.LRUCache(size)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	162 self.fs_cache = None
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	163 self._last_access_hit = None
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	164 self._invalidated_fs_items = set()
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	165
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	166 @property
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	167 def last_access_hit(self):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	168 return self._last_access_hit
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	169
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	170 def invalidate(self, key):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	171 logger.debug("Invalidating cache item '%s'." % key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	172 self.cache.invalidate(key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	173 if self.fs_cache:
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	174 logger.debug("Invalidating FS cache item '%s'." % key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	175 fs_key = _make_fs_cache_key(key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	176 self._invalidated_fs_items.add(fs_key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	177
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	178 def put(self, key, item, save_to_fs=True):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	179 self.cache.put(key, item)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	180 if self.fs_cache and save_to_fs:
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	181 fs_key = _make_fs_cache_key(key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	182 item_raw = json.dumps(item)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	183 self.fs_cache.write(fs_key, item_raw)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	184
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	185 def get(self, key, item_maker, fs_cache_time=None, save_to_fs=True):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	186 self._last_access_hit = True
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	187 item = self.cache.get(key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	188 if item is None:
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	189 if (self.fs_cache is not None and
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	190 fs_cache_time is not None):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	191 # Try first from the file-system cache.
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	192 fs_key = _make_fs_cache_key(key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	193 if (fs_key not in self._invalidated_fs_items and
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	194 self.fs_cache.isValid(fs_key, fs_cache_time)):
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	195 logger.debug("'%s' found in file-system cache." %
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	196 key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	197 item_raw = self.fs_cache.read(fs_key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	198 item = json.loads(
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	199 item_raw,
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	200 object_pairs_hook=collections.OrderedDict)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	201 self.cache.put(key, item)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	202 return item
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	203
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	204 # Look into the mem-cache.
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	205 logger.debug("'%s' not found in cache, must build." % key)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	206 item = item_maker()
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	207 self.cache.put(key, item)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	208 self._last_access_hit = False
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	209
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	210 # Save to the file-system if needed.
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	211 if self.fs_cache is not None and save_to_fs:
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	212 item_raw = json.dumps(item)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	213 self.fs_cache.write(fs_key, item_raw)
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	214
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	215 return item
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	216
ff6cc43fb40c internal: Move `MemCache` to the `cache` module, remove threading locks. Ludovic Chabant <ludovic@chabant.com> parents: 371 diff changeset	217

Mercurial > piecrust2

annotate piecrust/cache.py @ 550:6f216c1ab6b1