annotate piecrust/configuration.py @ 369:4b1019bb2533

serve: Giant refactor to change how we handle data when serving pages. * We need a distinction between source metadata and route metadata. In most cases they're the same, but in cases like taxonomy pages, route metadata contains more things that can't be in source metadata if we want to re-use cached pages. * Create a new `QualifiedPage` type which is a page with a specific route and route metadata. Pass this around in many places. * Instead of passing an URL around, use the route in the `QualifiedPage` to generate URLs. This is better since it removes the guess-work from trying to generate URLs for sub-pages. * Deep-copy app and page configurations before passing them around to things that could modify them, like data builders and such. * Exclude taxonomy pages from iterator data providers. * Properly nest iterator data providers for when the theme and user page sources are merged inside `site.pages`.
author Ludovic Chabant <ludovic@chabant.com>
date Sun, 03 May 2015 18:47:10 -0700
parents 734f2abf361c
children bdeeee777f85
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import re
367
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
2 import copy
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
3 import logging
67
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
4 import collections
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import yaml
67
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
6 from yaml.constructor import ConstructorError
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
9 logger = logging.getLogger(__name__)
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
10
367
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
11 default_allowed_types = (dict, list, tuple, int, bool, str)
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
12
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
13
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
14 class ConfigurationError(Exception):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
15 pass
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
16
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
17
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 class Configuration(object):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19 def __init__(self, values=None, validate=True):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 if values is not None:
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
21 self.setAll(values, validate)
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
22 else:
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
23 self._values = None
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24
138
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
25 def __contains__(self, key):
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
26 return self.has(key)
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
27
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
28 def __getitem__(self, key):
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
29 value = self.get(key)
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
30 if value is None:
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
31 raise KeyError()
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
32 return value
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
33
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
34 def __setitem__(self, key, value):
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
35 return self.set(key, value)
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
36
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
37 def setAll(self, values, validate=True):
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 if validate:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 self._validateAll(values)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 self._values = values
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41
367
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
42 def getDeepcopy(self, validate_types=False):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
43 if validate_types:
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
44 self.validateTypes()
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
45 return copy.deepcopy(self.get())
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
46
204
f98451237371 internal: Add ability to get a default value if a config value doesn't exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 138
diff changeset
47 def get(self, key_path=None, default_value=None):
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 self._ensureLoaded()
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 if key_path is None:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 return self._values
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 bits = key_path.split('/')
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 cur = self._values
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 for b in bits:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 cur = cur.get(b)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 if cur is None:
204
f98451237371 internal: Add ability to get a default value if a config value doesn't exist.
Ludovic Chabant <ludovic@chabant.com>
parents: 138
diff changeset
56 return default_value
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 return cur
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 def set(self, key_path, value):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60 self._ensureLoaded()
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61 value = self._validateValue(key_path, value)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62 bits = key_path.split('/')
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63 bitslen = len(bits)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 cur = self._values
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 for i, b in enumerate(bits):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66 if i == bitslen - 1:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67 cur[b] = value
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68 else:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 if b not in cur:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70 cur[b] = {}
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 cur = cur[b]
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 def has(self, key_path):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 self._ensureLoaded()
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 bits = key_path.split('/')
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 cur = self._values
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 for b in bits:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 cur = cur.get(b)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 if cur is None:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 return False
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81 return True
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83 def merge(self, other):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84 self._ensureLoaded()
138
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
85
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
86 if isinstance(other, dict):
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
87 other_values = other
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
88 elif isinstance(other, Configuration):
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
89 other_values = other._values
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
90 else:
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
91 raise Exception(
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
92 "Unsupported value type to merge: %s" % type(other))
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
93
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
94 merge_dicts(self._values, other_values,
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
95 validator=self._validateValue)
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
96
367
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
97 def validateTypes(self, allowed_types=default_allowed_types):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
98 self._validateDictTypesRecursive(self._values, allowed_types)
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
99
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
100 def _validateDictTypesRecursive(self, d, allowed_types):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
101 for k, v in d.items():
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
102 if not isinstance(k, str):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
103 raise ConfigurationError("Key '%s' is not a string." % k)
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
104 self._validateTypeRecursive(v, allowed_types)
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
105
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
106 def _validateListTypesRecursive(self, l, allowed_types):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
107 for v in l:
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
108 self._validateTypeRecursive(v, allowed_types)
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
109
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
110 def _validateTypeRecursive(self, v, allowed_types):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
111 if v is None:
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
112 return
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
113 if not isinstance(v, allowed_types):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
114 raise ConfigurationError(
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
115 "Value '%s' is of forbidden type: %s" % (v, type(v)))
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
116 if isinstance(v, dict):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
117 self._validateDictTypesRecursive(v, allowed_types)
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
118 elif isinstance(v, list):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
119 self._validateListTypesRecursive(v, allowed_types)
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
120
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
121 def _ensureLoaded(self):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
122 if self._values is None:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
123 self._load()
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
124
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
125 def _load(self):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
126 self._values = self._validateAll({})
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
127
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
128 def _validateAll(self, values):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
129 return values
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
130
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
131 def _validateValue(self, key_path, value):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
132 return value
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
133
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
134
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
135 def merge_dicts(source, merging, validator=None, *args):
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
136 if validator is None:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
137 validator = lambda k, v: v
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
138 _recurse_merge_dicts(source, merging, None, validator)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
139 for other in args:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
140 _recurse_merge_dicts(source, other, None, validator)
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
141
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
142
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
143 def _recurse_merge_dicts(local_cur, incoming_cur, parent_path, validator):
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
144 for k, v in incoming_cur.items():
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
145 key_path = k
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
146 if parent_path is not None:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
147 key_path = parent_path + '/' + k
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
148
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
149 local_v = local_cur.get(k)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
150 if local_v is not None:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
151 if isinstance(v, dict) and isinstance(local_v, dict):
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
152 _recurse_merge_dicts(local_v, v, key_path, validator)
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
153 elif isinstance(v, list) and isinstance(local_v, list):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
154 local_cur[k] = v + local_v
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
155 else:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
156 local_cur[k] = validator(key_path, v)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
157 else:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
158 local_cur[k] = validator(key_path, v)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
159
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
160
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
161 header_regex = re.compile(
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
162 r'(---\s*\n)(?P<header>(.*\n)*?)^(---\s*\n)', re.MULTILINE)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
163
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
164
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
165 def parse_config_header(text):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
166 m = header_regex.match(text)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
167 if m is not None:
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
168 header = str(m.group('header'))
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
169 config = yaml.load(header, Loader=ConfigurationLoader)
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
170 offset = m.end()
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
171 else:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
172 config = {}
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
173 offset = 0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
174 return config, offset
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
175
67
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
176
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
177 class ConfigurationLoader(yaml.SafeLoader):
67
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
178 """ A YAML loader that loads mappings into ordered dictionaries.
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
179 """
81
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
180 def __init__(self, *args, **kwargs):
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
181 super(ConfigurationLoader, self).__init__(*args, **kwargs)
81
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
182
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
183 self.add_constructor('tag:yaml.org,2002:map',
81
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
184 type(self).construct_yaml_map)
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
185 self.add_constructor('tag:yaml.org,2002:omap',
81
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
186 type(self).construct_yaml_map)
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
187 self.add_constructor('tag:yaml.org,2002:sexagesimal',
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
188 type(self).construct_yaml_time)
81
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
189
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
190 def construct_yaml_map(self, node):
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
191 data = collections.OrderedDict()
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
192 yield data
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
193 value = self.construct_mapping(node)
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
194 data.update(value)
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
195
67
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
196 def construct_mapping(self, node, deep=False):
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
197 if not isinstance(node, yaml.MappingNode):
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
198 raise ConstructorError(None, None,
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
199 "expected a mapping node, but found %s" % node.id,
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
200 node.start_mark)
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
201 mapping = collections.OrderedDict()
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
202 for key_node, value_node in node.value:
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
203 key = self.construct_object(key_node, deep=deep)
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
204 if not isinstance(key, collections.Hashable):
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
205 raise ConstructorError("while constructing a mapping", node.start_mark,
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
206 "found unhashable key", key_node.start_mark)
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
207 value = self.construct_object(value_node, deep=deep)
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
208 mapping[key] = value
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
209 return mapping
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
210
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
211 time_regexp = re.compile(
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
212 r'''^(?P<hour>[0-9][0-9]?)
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
213 :(?P<minute>[0-9][0-9])
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
214 (:(?P<second>[0-9][0-9])
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
215 (\.(?P<fraction>[0-9]+))?)?$''', re.X)
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
216
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
217 def construct_yaml_time(self, node):
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
218 self.construct_scalar(node)
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
219 match = self.time_regexp.match(node.value)
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
220 values = match.groupdict()
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
221 hour = int(values['hour'])
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
222 minute = int(values['minute'])
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
223 second = 0
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
224 if values['second']:
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
225 second = int(values['second'])
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
226 usec = 0
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
227 if values['fraction']:
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
228 usec = float('0.' + values['fraction'])
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
229 return second + minute * 60 + hour * 60 * 60 + usec
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
230
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
231
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
232 ConfigurationLoader.add_implicit_resolver(
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
233 'tag:yaml.org,2002:sexagesimal',
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
234 re.compile(r'''^[0-9][0-9]?:[0-9][0-9]
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
235 (:[0-9][0-9](\.[0-9]+)?)?$''', re.X),
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
236 list('0123456789'))
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
237
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
238
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
239 # We need to add our `sexagesimal` resolver before the `int` one, which
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
240 # already supports sexagesimal notation in YAML 1.1 (but not 1.2). However,
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
241 # because we know we pretty much always want it for representing time, we
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
242 # need a simple `12:30` to mean 45000, not 750. So that's why we override
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
243 # the default behaviour.
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
244 for ch in list('0123456789'):
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
245 ch_resolvers = ConfigurationLoader.yaml_implicit_resolvers[ch]
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
246 ch_resolvers.insert(0, ch_resolvers.pop())
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
247
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
248
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
249 class ConfigurationDumper(yaml.SafeDumper):
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
250 def represent_ordered_dict(self, data):
301
45aba3cb7228 config: Make YAML consider `omap` structures as normal maps.
Ludovic Chabant <ludovic@chabant.com>
parents: 204
diff changeset
251 # Not a typo: we're using `map` and not `omap` because we don't want
45aba3cb7228 config: Make YAML consider `omap` structures as normal maps.
Ludovic Chabant <ludovic@chabant.com>
parents: 204
diff changeset
252 # ugly type tags printed in the generated YAML markup, and because
45aba3cb7228 config: Make YAML consider `omap` structures as normal maps.
Ludovic Chabant <ludovic@chabant.com>
parents: 204
diff changeset
253 # we always load maps into `OrderedDicts` anyway.
45aba3cb7228 config: Make YAML consider `omap` structures as normal maps.
Ludovic Chabant <ludovic@chabant.com>
parents: 204
diff changeset
254 return self.represent_mapping('tag:yaml.org,2002:map', data)
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
255
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
256
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
257 ConfigurationDumper.add_representer(collections.OrderedDict,
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
258 ConfigurationDumper.represent_ordered_dict)
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
259