Mercurial > piecrust2

annotate piecrust/configuration.py @ 550:6f216c1ab6b1

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Add a flag to know which record entries got collapsed from last run. This makes it possible to find entries for things that were actually baked during the current run, as opposed to skipped because they were "clean".
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 04 Aug 2015 21:22:30 -0700
parents 1359b2b0cc73
children 9ccc933ac2c7
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import re
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
2 import logging
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
3 import collections.abc
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import yaml
67
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
5 from yaml.constructor import ConstructorError
444
1359b2b0cc73 performance: Use the fast YAML loader if available.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
6 try:
1359b2b0cc73 performance: Use the fast YAML loader if available.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
7 from yaml import CSafeLoader as SafeLoader
1359b2b0cc73 performance: Use the fast YAML loader if available.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
8 except ImportError:
1359b2b0cc73 performance: Use the fast YAML loader if available.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
9 from yaml import SafeLoader
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
12 logger = logging.getLogger(__name__)
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
13
431
bdeeee777f85 internal: Floats are also allowed in configurations, duh.
Ludovic Chabant <ludovic@chabant.com>
parents: 367
diff changeset
14 default_allowed_types = (dict, list, tuple, float, int, bool, str)
367
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
15
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
16
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
17 class ConfigurationError(Exception):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
18 pass
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
19
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
20
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
21 class Configuration(collections.abc.MutableMapping):
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22 def __init__(self, values=None, validate=True):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 if values is not None:
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
24 self.setAll(values, validate=validate)
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
25 else:
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
26 self._values = None
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27
138
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
28 def __getitem__(self, key):
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
29 self._ensureLoaded()
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
30 bits = key.split('/')
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
31 cur = self._values
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
32 for b in bits:
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
33 try:
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
34 cur = cur[b]
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
35 except KeyError:
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
36 raise KeyError("No such item: %s" % key)
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
37 return cur
138
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
38
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
39 def __setitem__(self, key, value):
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 self._ensureLoaded()
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
41 value = self._validateValue(key, value)
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
42 bits = key.split('/')
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 bitslen = len(bits)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 cur = self._values
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 for i, b in enumerate(bits):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 if i == bitslen - 1:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 cur[b] = value
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 else:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 if b not in cur:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 cur[b] = {}
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 cur = cur[b]
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
53 def __delitem__(self, key):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
54 raise NotImplementedError()
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
55
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
56 def __iter__(self):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
57 self._ensureLoaded()
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
58 return iter(self._values)
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
59
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
60 def __len__(self):
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61 self._ensureLoaded()
440
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
62 return len(self._values)
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
63
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
64 def has(self, key):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
65 return key in self
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
66
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
67 def set(self, key, value):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
68 self[key] = value
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
69
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
70 def setAll(self, values, validate=False):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
71 if validate:
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
72 self._validateAll(values)
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
73 self._values = values
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
74
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
75 def getAll(self):
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
76 self._ensureLoaded()
32c7c2d219d2 performance: Refactor how data is managed to reduce copying.
Ludovic Chabant <ludovic@chabant.com>
parents: 431
diff changeset
77 return self._values
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 def merge(self, other):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 self._ensureLoaded()
138
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
81
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
82 if isinstance(other, dict):
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
83 other_values = other
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
84 elif isinstance(other, Configuration):
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
85 other_values = other._values
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
86 else:
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
87 raise Exception(
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
88 "Unsupported value type to merge: %s" % type(other))
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
89
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
90 merge_dicts(self._values, other_values,
b540d431f2da Make configuration class more like `dict`, add support for merging `dicts`.
Ludovic Chabant <ludovic@chabant.com>
parents: 107
diff changeset
91 validator=self._validateValue)
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
92
367
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
93 def validateTypes(self, allowed_types=default_allowed_types):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
94 self._validateDictTypesRecursive(self._values, allowed_types)
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
95
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
96 def _validateDictTypesRecursive(self, d, allowed_types):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
97 for k, v in d.items():
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
98 if not isinstance(k, str):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
99 raise ConfigurationError("Key '%s' is not a string." % k)
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
100 self._validateTypeRecursive(v, allowed_types)
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
101
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
102 def _validateListTypesRecursive(self, l, allowed_types):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
103 for v in l:
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
104 self._validateTypeRecursive(v, allowed_types)
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
105
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
106 def _validateTypeRecursive(self, v, allowed_types):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
107 if v is None:
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
108 return
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
109 if not isinstance(v, allowed_types):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
110 raise ConfigurationError(
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
111 "Value '%s' is of forbidden type: %s" % (v, type(v)))
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
112 if isinstance(v, dict):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
113 self._validateDictTypesRecursive(v, allowed_types)
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
114 elif isinstance(v, list):
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
115 self._validateListTypesRecursive(v, allowed_types)
734f2abf361c config: Add method to deep-copy a config and validate its contents.
Ludovic Chabant <ludovic@chabant.com>
parents: 301
diff changeset
116
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
117 def _ensureLoaded(self):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
118 if self._values is None:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
119 self._load()
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
120
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
121 def _load(self):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
122 self._values = self._validateAll({})
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
123
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
124 def _validateAll(self, values):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
125 return values
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
126
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
127 def _validateValue(self, key_path, value):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
128 return value
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
129
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
130
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
131 def merge_dicts(source, merging, validator=None, *args):
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
132 if validator is None:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
133 validator = lambda k, v: v
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
134 _recurse_merge_dicts(source, merging, None, validator)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
135 for other in args:
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 2
diff changeset
136 _recurse_merge_dicts(source, other, None, validator)
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
137
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
138
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
139 def _recurse_merge_dicts(local_cur, incoming_cur, parent_path, validator):
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
140 for k, v in incoming_cur.items():
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
141 key_path = k
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
142 if parent_path is not None:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
143 key_path = parent_path + '/' + k
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
144
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
145 local_v = local_cur.get(k)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
146 if local_v is not None:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
147 if isinstance(v, dict) and isinstance(local_v, dict):
2
40fa08b261b9 Added unit tests (using `py.test`) for `Configuration`.
Ludovic Chabant <ludovic@chabant.com>
parents: 0
diff changeset
148 _recurse_merge_dicts(local_v, v, key_path, validator)
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
149 elif isinstance(v, list) and isinstance(local_v, list):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
150 local_cur[k] = v + local_v
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
151 else:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
152 local_cur[k] = validator(key_path, v)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
153 else:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
154 local_cur[k] = validator(key_path, v)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
155
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
156
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
157 header_regex = re.compile(
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
158 r'(---\s*\n)(?P<header>(.*\n)*?)^(---\s*\n)', re.MULTILINE)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
159
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
160
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
161 def parse_config_header(text):
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
162 m = header_regex.match(text)
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
163 if m is not None:
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
164 header = str(m.group('header'))
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
165 config = yaml.load(header, Loader=ConfigurationLoader)
0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
166 offset = m.end()
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
167 else:
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
168 config = {}
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
169 offset = 0
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
170 return config, offset
a212a3f2e3ee Initial commit.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
171
67
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
172
444
1359b2b0cc73 performance: Use the fast YAML loader if available.
Ludovic Chabant <ludovic@chabant.com>
parents: 440
diff changeset
173 class ConfigurationLoader(SafeLoader):
67
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
174 """ A YAML loader that loads mappings into ordered dictionaries.
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
175 """
81
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
176 def __init__(self, *args, **kwargs):
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
177 super(ConfigurationLoader, self).__init__(*args, **kwargs)
81
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
178
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
179 self.add_constructor('tag:yaml.org,2002:map',
81
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
180 type(self).construct_yaml_map)
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
181 self.add_constructor('tag:yaml.org,2002:omap',
81
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
182 type(self).construct_yaml_map)
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
183 self.add_constructor('tag:yaml.org,2002:sexagesimal',
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
184 type(self).construct_yaml_time)
81
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
185
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
186 def construct_yaml_map(self, node):
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
187 data = collections.OrderedDict()
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
188 yield data
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
189 value = self.construct_mapping(node)
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
190 data.update(value)
d64e4703f5e6 Propertly create `OrderedDict`s when loading YAML.
Ludovic Chabant <ludovic@chabant.com>
parents: 68
diff changeset
191
67
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
192 def construct_mapping(self, node, deep=False):
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
193 if not isinstance(node, yaml.MappingNode):
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
194 raise ConstructorError(None, None,
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
195 "expected a mapping node, but found %s" % node.id,
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
196 node.start_mark)
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
197 mapping = collections.OrderedDict()
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
198 for key_node, value_node in node.value:
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
199 key = self.construct_object(key_node, deep=deep)
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
200 if not isinstance(key, collections.Hashable):
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
201 raise ConstructorError("while constructing a mapping", node.start_mark,
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
202 "found unhashable key", key_node.start_mark)
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
203 value = self.construct_object(value_node, deep=deep)
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
204 mapping[key] = value
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
205 return mapping
563ce5dd02af I don't care what the YAML spec says, ordered maps are the only sane way.
Ludovic Chabant <ludovic@chabant.com>
parents: 5
diff changeset
206
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
207 time_regexp = re.compile(
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
208 r'''^(?P<hour>[0-9][0-9]?)
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
209 :(?P<minute>[0-9][0-9])
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
210 (:(?P<second>[0-9][0-9])
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
211 (\.(?P<fraction>[0-9]+))?)?$''', re.X)
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
212
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
213 def construct_yaml_time(self, node):
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
214 self.construct_scalar(node)
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
215 match = self.time_regexp.match(node.value)
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
216 values = match.groupdict()
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
217 hour = int(values['hour'])
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
218 minute = int(values['minute'])
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
219 second = 0
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
220 if values['second']:
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
221 second = int(values['second'])
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
222 usec = 0
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
223 if values['fraction']:
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
224 usec = float('0.' + values['fraction'])
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
225 return second + minute * 60 + hour * 60 * 60 + usec
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
226
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
227
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
228 ConfigurationLoader.add_implicit_resolver(
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
229 'tag:yaml.org,2002:sexagesimal',
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
230 re.compile(r'''^[0-9][0-9]?:[0-9][0-9]
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
231 (:[0-9][0-9](\.[0-9]+)?)?$''', re.X),
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
232 list('0123456789'))
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
233
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
234
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
235 # We need to add our `sexagesimal` resolver before the `int` one, which
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
236 # already supports sexagesimal notation in YAML 1.1 (but not 1.2). However,
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
237 # because we know we pretty much always want it for representing time, we
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
238 # need a simple `12:30` to mean 45000, not 750. So that's why we override
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
239 # the default behaviour.
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
240 for ch in list('0123456789'):
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
241 ch_resolvers = ConfigurationLoader.yaml_implicit_resolvers[ch]
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
242 ch_resolvers.insert(0, ch_resolvers.pop())
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
243
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
244
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
245 class ConfigurationDumper(yaml.SafeDumper):
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
246 def represent_ordered_dict(self, data):
301
45aba3cb7228 config: Make YAML consider `omap` structures as normal maps.
Ludovic Chabant <ludovic@chabant.com>
parents: 204
diff changeset
247 # Not a typo: we're using `map` and not `omap` because we don't want
45aba3cb7228 config: Make YAML consider `omap` structures as normal maps.
Ludovic Chabant <ludovic@chabant.com>
parents: 204
diff changeset
248 # ugly type tags printed in the generated YAML markup, and because
45aba3cb7228 config: Make YAML consider `omap` structures as normal maps.
Ludovic Chabant <ludovic@chabant.com>
parents: 204
diff changeset
249 # we always load maps into `OrderedDicts` anyway.
45aba3cb7228 config: Make YAML consider `omap` structures as normal maps.
Ludovic Chabant <ludovic@chabant.com>
parents: 204
diff changeset
250 return self.represent_mapping('tag:yaml.org,2002:map', data)
107
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
251
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
252
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
253 ConfigurationDumper.add_representer(collections.OrderedDict,
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
254 ConfigurationDumper.represent_ordered_dict)
10fc9c8bf682 Better support for times in YAML interop.
Ludovic Chabant <ludovic@chabant.com>
parents: 81
diff changeset
255