Mercurial > piecrust2

annotate tests/test_fastpickle.py @ 1188:a7c43131d871

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 15 Jun 2021 22:36:23 -0700
parents 8adc27285d93
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 717
diff changeset
1 import io
450
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import datetime
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import pytest
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 717
diff changeset
4 from piecrust.fastpickle import pickle, unpickle, pickle_intob, unpickle_fromb
450
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7 class Foo(object):
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 def __init__(self, name):
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 self.name = name
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10 self.bars = []
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13 class Bar(object):
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 def __init__(self, value):
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 self.value = value
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 @pytest.mark.parametrize(
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19 'obj, expected',
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 [
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 (True, True),
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22 (42, 42),
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 (3.14, 3.14),
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 (datetime.date(2015, 5, 21), datetime.date(2015, 5, 21)),
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 (datetime.datetime(2015, 5, 21, 12, 55, 32),
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 datetime.datetime(2015, 5, 21, 12, 55, 32)),
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 (datetime.time(9, 25, 57), datetime.time(9, 25, 57)),
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 ((1, 2, 3), (1, 2, 3)),
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29 ([1, 2, 3], [1, 2, 3]),
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 ({'foo': 1, 'bar': 2}, {'foo': 1, 'bar': 2}),
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 (set([1, 2, 3]), set([1, 2, 3])),
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 ({'foo': [1, 2, 3], 'bar': {'one': 1, 'two': 2}},
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 {'foo': [1, 2, 3], 'bar': {'one': 1, 'two': 2}})
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 ])
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35 def test_pickle_unpickle(obj, expected):
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 data = pickle(obj)
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 actual = unpickle(data)
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 assert actual == expected
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 717
diff changeset
40 with io.BytesIO() as buf:
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 717
diff changeset
41 pickle_intob(obj, buf)
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 717
diff changeset
42 size = buf.tell()
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 717
diff changeset
43 buf.seek(0)
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 717
diff changeset
44 actual = unpickle_fromb(buf, size)
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 717
diff changeset
45 assert actual == expected
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 717
diff changeset
46
450
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 def test_objects():
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 f = Foo('foo')
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 f.bars.append(Bar(1))
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 f.bars.append(Bar(2))
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 data = pickle(f)
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 o = unpickle(data)
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56 assert type(o) == Foo
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 assert o.name == 'foo'
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 assert len(o.bars) == 2
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 for i in range(2):
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60 assert f.bars[i].value == o.bars[i].value
298f8f46432a internal: Add a `fastpickle` module to help with multiprocess serialization.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61
717
fcfbe103cfd1 internal: Fix some bugs with the `fastpickle` module.
Ludovic Chabant <ludovic@chabant.com>
parents: 450
diff changeset
62
fcfbe103cfd1 internal: Fix some bugs with the `fastpickle` module.
Ludovic Chabant <ludovic@chabant.com>
parents: 450
diff changeset
63 def test_reentrance():
fcfbe103cfd1 internal: Fix some bugs with the `fastpickle` module.
Ludovic Chabant <ludovic@chabant.com>
parents: 450
diff changeset
64 a = {'test_ints': 42, 'test_set': set([1, 2])}
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 717
diff changeset
65 data = pickle(a)
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 717
diff changeset
66 b = unpickle(data)
717
fcfbe103cfd1 internal: Fix some bugs with the `fastpickle` module.
Ludovic Chabant <ludovic@chabant.com>
parents: 450
diff changeset
67 assert a == b
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 717
diff changeset
68 other_b = unpickle(data)
717
fcfbe103cfd1 internal: Fix some bugs with the `fastpickle` module.
Ludovic Chabant <ludovic@chabant.com>
parents: 450
diff changeset
69 assert a == other_b
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 717
diff changeset
70 c = unpickle(data)
717
fcfbe103cfd1 internal: Fix some bugs with the `fastpickle` module.
Ludovic Chabant <ludovic@chabant.com>
parents: 450
diff changeset
71 assert a == c