Mercurial > piecrust2
annotate piecrust/cache.py @ 411:e7b865f8f335
bake: Enable multiprocess baking.
Baking is now done by running a worker per CPU, and sending jobs to them.
This changes several things across the codebase:
* Ability to not cache things related to pages other than the 'main' page
(i.e. the page at the bottom of the execution stack).
* Decouple the baking process from the bake records, so only the main process
keeps track (and modifies) the bake record.
* Remove the need for 'batch page getters' and loading a page directly from
the page factories.
There are various smaller changes too included here, including support for
scope performance timers that are saved with the bake record and can be
printed out to the console. Yes I got carried away.
For testing, the in-memory 'mock' file-system doesn't work anymore, since
we're spawning processes, so this is replaced by a 'tmpfs' file-system which
is saved in temporary files on disk and deleted after tests have run.
author | Ludovic Chabant <ludovic@chabant.com> |
---|---|
date | Fri, 12 Jun 2015 17:09:19 -0700 |
parents | c2ca72fb7f0b |
children | ff6cc43fb40c |
rev | line source |
---|---|
0 | 1 import os |
2 import os.path | |
371
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
3 import shutil |
0 | 4 import codecs |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
5 import logging |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
6 import threading |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
7 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
8 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
9 logger = logging.getLogger(__name__) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
10 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
11 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
12 class ExtensibleCache(object): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
13 def __init__(self, base_dir): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
14 self.base_dir = base_dir |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
15 self.lock = threading.Lock() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
16 self.caches = {} |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
17 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
18 @property |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
19 def enabled(self): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
20 return True |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
21 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
22 def getCache(self, name): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
23 c = self.caches.get(name) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
24 if c is None: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
25 with self.lock: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
26 c = self.caches.get(name) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
27 if c is None: |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
28 c_dir = os.path.join(self.base_dir, name) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
29 if not os.path.isdir(c_dir): |
5 | 30 os.makedirs(c_dir, 0o755) |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
31 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
32 c = SimpleCache(c_dir) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
33 self.caches[name] = c |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
34 return c |
0 | 35 |
45
efd0d3bacc9e
Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
5
diff
changeset
|
36 def getCacheDir(self, name): |
efd0d3bacc9e
Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
5
diff
changeset
|
37 return os.path.join(self.base_dir, name) |
efd0d3bacc9e
Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
5
diff
changeset
|
38 |
105
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
45
diff
changeset
|
39 def getCacheNames(self, except_names=None): |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
45
diff
changeset
|
40 _, dirnames, __ = next(os.walk(self.base_dir)) |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
45
diff
changeset
|
41 if except_names is None: |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
45
diff
changeset
|
42 return dirnames |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
45
diff
changeset
|
43 return [dn for dn in dirnames if dn not in except_names] |
7d2fdf43d7ca
Property clean all caches when force baking, except the `app` cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
45
diff
changeset
|
44 |
371
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
45 def clearCache(self, name): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
46 cache_dir = self.getCacheDir(name) |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
47 if os.path.isdir(cache_dir): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
48 logger.debug("Cleaning cache: %s" % cache_dir) |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
49 shutil.rmtree(cache_dir) |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
50 |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
51 def clearCaches(self, except_names=None): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
52 for name in self.getCacheNames(except_names=except_names): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
53 self.clearCache(name) |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
54 |
0 | 55 |
56 class SimpleCache(object): | |
57 def __init__(self, base_dir): | |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
58 self.base_dir = base_dir |
0 | 59 if not os.path.isdir(base_dir): |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
60 raise Exception("Cache directory doesn't exist: %s" % base_dir) |
0 | 61 |
62 def isValid(self, path, time): | |
63 cache_time = self.getCacheTime(path) | |
64 if cache_time is None: | |
65 return False | |
66 if isinstance(time, list): | |
67 for t in time: | |
68 if cache_time < t: | |
69 return False | |
70 return True | |
71 return cache_time >= time | |
72 | |
73 def getCacheTime(self, path): | |
74 cache_path = self.getCachePath(path) | |
75 try: | |
76 return os.path.getmtime(cache_path) | |
77 except os.error: | |
78 return None | |
79 | |
80 def has(self, path): | |
81 cache_path = self.getCachePath(path) | |
82 return os.path.isfile(cache_path) | |
83 | |
84 def read(self, path): | |
85 cache_path = self.getCachePath(path) | |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
86 logger.debug("Reading cache: %s" % cache_path) |
0 | 87 with codecs.open(cache_path, 'r', 'utf-8') as fp: |
88 return fp.read() | |
89 | |
90 def write(self, path, content): | |
91 cache_path = self.getCachePath(path) | |
92 cache_dir = os.path.dirname(cache_path) | |
93 if not os.path.isdir(cache_dir): | |
5 | 94 os.makedirs(cache_dir, 0o755) |
45
efd0d3bacc9e
Don't recursively clean the cache.
Ludovic Chabant <ludovic@chabant.com>
parents:
5
diff
changeset
|
95 logger.debug("Writing cache: %s" % cache_path) |
0 | 96 with codecs.open(cache_path, 'w', 'utf-8') as fp: |
97 fp.write(content) | |
98 | |
3
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
99 def getCachePath(self, path): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
100 if path.startswith('.'): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
101 path = '__index__' + path |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
102 return os.path.join(self.base_dir, path) |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
103 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
104 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
105 class NullCache(object): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
106 def isValid(self, path, time): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
107 return False |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
108 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
109 def getCacheTime(self, path): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
110 return None |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
111 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
112 def has(self, path): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
113 return False |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
114 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
115 def read(self, path): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
116 raise Exception("Null cache has no data.") |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
117 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
118 def write(self, path, content): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
119 pass |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
120 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
121 def getCachePath(self, path): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
122 raise Exception("Null cache can't make paths.") |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
123 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
124 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
125 class NullExtensibleCache(object): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
126 def __init__(self): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
127 self.null_cache = NullCache() |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
128 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
129 @property |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
130 def enabled(self): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
131 return False |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
132 |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
133 def getCache(self, name): |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
134 return self.null_cache |
f485ba500df3
Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
0
diff
changeset
|
135 |
371
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
136 def getCacheDir(self, name): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
137 raise NotImplementedError() |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
138 |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
139 def getCacheNames(self, except_names=None): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
140 return [] |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
141 |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
142 def clearCache(self, name): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
143 pass |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
144 |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
145 def clearCaches(self, except_names=None): |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
146 pass |
c2ca72fb7f0b
caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents:
105
diff
changeset
|
147 |