Mercurial > piecrust2

annotate piecrust/data/providersdata.py @ 1188:a7c43131d871

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 15 Jun 2021 22:36:23 -0700
parents ed308313bcda
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import re
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import collections.abc
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
3 from piecrust.configuration import ConfigurationError
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
4 from piecrust.dataproviders.base import (
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
5 DataProvider, build_data_provider)
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 re_endpoint_sep = re.compile(r'[\/\.]')
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11 class DataProvidersData(collections.abc.Mapping):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12 def __init__(self, page):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13 self._page = page
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 self._dict = None
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16 def __getitem__(self, name):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 self._load()
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 return self._dict[name]
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 def __iter__(self):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 self._load()
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22 return iter(self._dict)
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 def __len__(self):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 self._load()
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 return len(self._dict)
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 def _load(self):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29 if self._dict is not None:
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 return
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 self._dict = {}
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
33 for source in self._page.app.sources:
979
45ad976712ec tests: Big push to get the tests to pass again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
34 pname = source.config.get('data_type') or 'page_iterator'
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
35 pendpoint = source.config.get('data_endpoint')
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
36 if not pname or not pendpoint:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
37 continue
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
38
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
39 endpoint_bits = re_endpoint_sep.split(pendpoint)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
40 endpoint = self._dict
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
41 for e in endpoint_bits[:-1]:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
42 if e not in endpoint:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
43 endpoint[e] = {}
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
44 endpoint = endpoint[e]
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
45 existing = endpoint.get(endpoint_bits[-1])
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
46
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
47 if existing is None:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
48 provider = build_data_provider(pname, source, self._page)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
49 endpoint[endpoint_bits[-1]] = provider
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
50 elif isinstance(existing, DataProvider):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
51 existing._addSource(source)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
52 else:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
53 raise ConfigurationError(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
54 "Endpoint '%s' can't be used for a data provider because "
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 847
diff changeset
55 "it's already used for something else." % pendpoint)