Mercurial > piecrust2

annotate piecrust/data/base.py @ 1188:a7c43131d871

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 15 Jun 2021 22:36:23 -0700
parents 501bd4ab7e06
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1010
501bd4ab7e06 internal: Remove unused parameter.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
1 import time
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
2 import collections.abc
381
4c9eab0e283b data: Fix problems with using non-existing metadata on a linked page.
Ludovic Chabant <ludovic@chabant.com>
parents: 369
diff changeset
3
4c9eab0e283b data: Fix problems with using non-existing metadata on a linked page.
Ludovic Chabant <ludovic@chabant.com>
parents: 369
diff changeset
4
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
5 class MergedMapping(collections.abc.Mapping):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
6 """ Provides a dictionary-like object that's really the aggregation of
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
7 multiple dictionary-like objects.
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 """
1010
501bd4ab7e06 internal: Remove unused parameter.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
9 def __init__(self, dicts, path=''):
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
10 self._dicts = dicts
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
11 self._path = path
226
e9dc18a275ff data: Add ability for `IPaginationSource`s to specify how to get settings.
Ludovic Chabant <ludovic@chabant.com>
parents: 158
diff changeset
12
12
30a42341cfa8 Define page slugs properly, avoid recursions with debug data.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
13 def __getattr__(self, name):
30a42341cfa8 Define page slugs properly, avoid recursions with debug data.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
14 try:
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
15 return self[name]
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
16 except KeyError:
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
17 raise AttributeError("No such attribute: %s" % self._subp(name))
12
30a42341cfa8 Define page slugs properly, avoid recursions with debug data.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
18
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19 def __getitem__(self, name):
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
20 values = []
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
21 for d in self._dicts:
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
22 try:
713
d446029c9478 data: Support both objects and dictionaries in `MergedMapping`.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
23 val = getattr(d, name)
d446029c9478 data: Support both objects and dictionaries in `MergedMapping`.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
24 values.append(val)
d446029c9478 data: Support both objects and dictionaries in `MergedMapping`.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
25 continue
d446029c9478 data: Support both objects and dictionaries in `MergedMapping`.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
26 except AttributeError:
d446029c9478 data: Support both objects and dictionaries in `MergedMapping`.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
27 pass
d446029c9478 data: Support both objects and dictionaries in `MergedMapping`.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
28
d446029c9478 data: Support both objects and dictionaries in `MergedMapping`.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
29 try:
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
30 val = d[name]
713
d446029c9478 data: Support both objects and dictionaries in `MergedMapping`.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
31 values.append(val)
d446029c9478 data: Support both objects and dictionaries in `MergedMapping`.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
32 continue
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
33 except KeyError:
713
d446029c9478 data: Support both objects and dictionaries in `MergedMapping`.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
34 pass
236
eaf18442bff8 internal: Add support for "wildcard" loader in `LazyPageConfigData`.
Ludovic Chabant <ludovic@chabant.com>
parents: 228
diff changeset
35
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
36 if len(values) == 0:
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
37 raise KeyError("No such item: %s" % self._subp(name))
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
38 if len(values) == 1:
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
39 return values[0]
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
41 for val in values:
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
42 if not isinstance(val, (dict, collections.abc.Mapping)):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
43 raise Exception(
901
1d5f02778723 internal: PEP8
Ludovic Chabant <ludovic@chabant.com>
parents: 713
diff changeset
44 "Template data for '%s' contains an incompatible mix "
1d5f02778723 internal: PEP8
Ludovic Chabant <ludovic@chabant.com>
parents: 713
diff changeset
45 "of data: %s" % (
1d5f02778723 internal: PEP8
Ludovic Chabant <ludovic@chabant.com>
parents: 713
diff changeset
46 self._subp(name),
1d5f02778723 internal: PEP8
Ludovic Chabant <ludovic@chabant.com>
parents: 713
diff changeset
47 ', '.join([str(type(v)) for v in values])))
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
49 return MergedMapping(values, self._subp(name))
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
51 def __iter__(self):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
52 keys = set()
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
53 for d in self._dicts:
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
54 keys |= set(d.keys())
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
55 return iter(keys)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
57 def __len__(self):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
58 keys = set()
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
59 for d in self._dicts:
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
60 keys |= set(d.keys())
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
61 return len(keys)
12
30a42341cfa8 Define page slugs properly, avoid recursions with debug data.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
62
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
63 def _subp(self, name):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
64 return '%s/%s' % (self._path, name)
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
65
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
66 def _prependMapping(self, d):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
67 self._dicts.insert(0, d)
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
68
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
69 def _appendMapping(self, d):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 433
diff changeset
70 self._dicts.append(d)
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
71