annotate tests/test_data_iterators.py @ 411:e7b865f8f335

bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.
author Ludovic Chabant <ludovic@chabant.com>
date Fri, 12 Jun 2015 17:09:19 -0700
parents 4379d8f8f831
children e35407c60e00
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
10
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import mock
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 from piecrust.data.iterators import PageIterator
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 from piecrust.page import Page, PageConfiguration
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 def test_skip():
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7 it = PageIterator(range(12))
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 it.skip(5)
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 assert it.total_count == 12
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10 assert len(it) == 7
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11 assert list(it) == list(range(5, 12))
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 def test_limit():
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 it = PageIterator(range(12))
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16 it.limit(4)
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 assert it.total_count == 12
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 assert len(it) == 4
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19 assert list(it) == list(range(4))
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22 def test_slice():
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 it = PageIterator(range(12))
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 it.slice(3, 4)
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 assert it.total_count == 12
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 assert len(it) == 4
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 assert list(it) == list(range(3, 7))
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 def test_natural_sort():
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 it = PageIterator([4, 3, 1, 2, 0])
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 it.sort()
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 assert it.total_count == 5
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 assert len(it) == 5
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35 assert list(it) == list(range(5))
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 def test_natural_sort_reversed():
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 it = PageIterator([4, 3, 1, 2, 0])
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 it.sort(reverse=True)
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 assert it.total_count == 5
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 assert len(it) == 5
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 assert list(it) == list(reversed(range(5)))
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 class TestItem(object):
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 def __init__(self, value):
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 self.name = str(value)
233
4379d8f8f831 internal: Removing some dependency of filters and iterators on pages.
Ludovic Chabant <ludovic@chabant.com>
parents: 10
diff changeset
49 self.foo = value
10
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 def __eq__(self, other):
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 return other.name == self.name
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 def test_setting_sort():
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56 it = PageIterator([TestItem(v) for v in [4, 3, 1, 2, 0]])
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 it.sort('foo')
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 assert it.total_count == 5
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 assert len(it) == 5
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60 assert list(it) == [TestItem(v) for v in range(5)]
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63 def test_setting_sort_reversed():
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 it = PageIterator([TestItem(v) for v in [4, 3, 1, 2, 0]])
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 it.sort('foo', reverse=True)
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66 assert it.total_count == 5
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67 assert len(it) == 5
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68 assert list(it) == [TestItem(v) for v in reversed(range(5))]
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 def test_filter():
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 page = mock.MagicMock(spec=Page)
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 page.config = PageConfiguration()
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 page.config.set('threes', {'is_foo': 3})
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 it = PageIterator([TestItem(v) for v in [3, 2, 3, 1, 4, 3]], page)
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 it.filter('threes')
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 assert it.total_count == 3
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 assert len(it) == 3
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 assert list(it) == [TestItem(3), TestItem(3), TestItem(3)]
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82 def test_magic_filter():
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83 it = PageIterator([TestItem(v) for v in [3, 2, 3, 1, 4, 3]])
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84 it.is_foo(3)
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
85 assert it.total_count == 3
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
86 assert len(it) == 3
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
87 assert list(it) == [TestItem(3), TestItem(3), TestItem(3)]
cd35d356ccce Fix some bugs with iterators, add some unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
88