Mercurial > piecrust2
annotate piecrust/cache.py @ 371:c2ca72fb7f0b 2.0.0a8
caching: Use separate caches for config variants and other contexts.
* The `_cache` directory is now organized in multiple "sub-caches" for
different contexts.
* A new context is created when config variants or overrides are applied.
* `serve` context uses a different context that the other commends, to prevent
the `bake` command's output from messing up the preview server (e.g. with
how asset URLs are generated differently between the two).
* Fix a few places where the cache directory was referenced directly.
author | Ludovic Chabant <ludovic@chabant.com> |
---|---|
date | Sun, 03 May 2015 23:59:46 -0700 |
parents | 7d2fdf43d7ca |
children | ff6cc43fb40c |
rev | line source |
---|---|
0 | 1 import os |
2 import os.path | |
371
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
3 import shutil |
0 | 4 import codecs |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
5 import logging |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
6 import threading |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
7 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
8 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
9 logger = logging.getLogger(__name__) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
10 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
11 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
12 class ExtensibleCache(object): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
13 def __init__(self, base_dir): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
14 self.base_dir = base_dir |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
15 self.lock = threading.Lock() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
16 self.caches = {} |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
17 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
18 @property |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
19 def enabled(self): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
20 return True |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
21 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
22 def getCache(self, name): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
23 c = self.caches.get(name) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
24 if c is None: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
25 with self.lock: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
26 c = self.caches.get(name) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
27 if c is None: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
28 c_dir = os.path.join(self.base_dir, name) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
29 if not os.path.isdir(c_dir): |
5 | 30 os.makedirs(c_dir, 0o755) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
31 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
32 c = SimpleCache(c_dir) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
33 self.caches[name] = c |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
34 return c |
0 | 35 |
45
efd0d3bacc9e
Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
5
diff
changeset
|
36 def getCacheDir(self, name): |
efd0d3bacc9e
Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
5
diff
changeset
|
37 return os.path.join(self.base_dir, name) |
efd0d3bacc9e
Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
5
diff
changeset
|
38 |
105
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
45
diff
changeset
|
39 def getCacheNames(self, except_names=None): |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
45
diff
changeset
|
40 _, dirnames, __ = next(os.walk(self.base_dir)) |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
45
diff
changeset
|
41 if except_names is None: |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
45
diff
changeset
|
42 return dirnames |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
45
diff
changeset
|
43 return [dn for dn in dirnames if dn not in except_names] |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
45
diff
changeset
|
44 |
371
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
45 def clearCache(self, name): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
46 cache_dir = self.getCacheDir(name) |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
47 if os.path.isdir(cache_dir): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
48 logger.debug("Cleaning cache: %s" % cache_dir) |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
49 shutil.rmtree(cache_dir) |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
50 |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
51 def clearCaches(self, except_names=None): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
52 for name in self.getCacheNames(except_names=except_names): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
53 self.clearCache(name) |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
54 |
0 | 55 |
56 class SimpleCache(object): | |
57 def __init__(self, base_dir): | |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
58 self.base_dir = base_dir |
0 | 59 if not os.path.isdir(base_dir): |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
60 raise Exception("Cache directory doesn't exist: %s" % base_dir) |
0 | 61 |
62 def isValid(self, path, time): | |
63 cache_time = self.getCacheTime(path) | |
64 if cache_time is None: | |
65 return False | |
66 if isinstance(time, list): | |
67 for t in time: | |
68 if cache_time < t: | |
69 return False | |
70 return True | |
71 return cache_time >= time | |
72 | |
73 def getCacheTime(self, path): | |
74 cache_path = self.getCachePath(path) | |
75 try: | |
76 return os.path.getmtime(cache_path) | |
77 except os.error: | |
78 return None | |
79 | |
80 def has(self, path): | |
81 cache_path = self.getCachePath(path) | |
82 return os.path.isfile(cache_path) | |
83 | |
84 def read(self, path): | |
85 cache_path = self.getCachePath(path) | |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
86 logger.debug("Reading cache: %s" % cache_path) |
0 | 87 with codecs.open(cache_path, 'r', 'utf-8') as fp: |
88 return fp.read() | |
89 | |
90 def write(self, path, content): | |
91 cache_path = self.getCachePath(path) | |
92 cache_dir = os.path.dirname(cache_path) | |
93 if not os.path.isdir(cache_dir): | |
5 | 94 os.makedirs(cache_dir, 0o755) |
45
efd0d3bacc9e
Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
5
diff
changeset
|
95 logger.debug("Writing cache: %s" % cache_path) |
0 | 96 with codecs.open(cache_path, 'w', 'utf-8') as fp: |
97 fp.write(content) | |
98 | |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
99 def getCachePath(self, path): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
100 if path.startswith('.'): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
101 path = '__index__' + path |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
102 return os.path.join(self.base_dir, path) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
103 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
104 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
105 class NullCache(object): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
106 def isValid(self, path, time): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
107 return False |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
108 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
109 def getCacheTime(self, path): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
110 return None |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
111 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
112 def has(self, path): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
113 return False |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
114 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
115 def read(self, path): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
116 raise Exception("Null cache has no data.") |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
117 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
118 def write(self, path, content): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
119 pass |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
120 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
121 def getCachePath(self, path): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
122 raise Exception("Null cache can't make paths.") |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
123 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
124 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
125 class NullExtensibleCache(object): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
126 def __init__(self): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
127 self.null_cache = NullCache() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
128 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
129 @property |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
130 def enabled(self): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
131 return False |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
132 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
133 def getCache(self, name): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
134 return self.null_cache |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
135 |
371
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
136 def getCacheDir(self, name): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
137 raise NotImplementedError() |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
138 |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
139 def getCacheNames(self, except_names=None): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
140 return [] |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
141 |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
142 def clearCache(self, name): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
143 pass |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
144 |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
145 def clearCaches(self, except_names=None): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
146 pass |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
147 |