Mercurial > piecrust2

annotate tests/test_data_linker.py @ 411:e7b865f8f335

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.
author Ludovic Chabant <ludovic@chabant.com>
date Fri, 12 Jun 2015 17:09:19 -0700
parents 27b10024f8d8
children f987b29d6fab
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
404
27b10024f8d8 linker: Add ability to return the parent and ancestors of a page.
Ludovic Chabant <ludovic@chabant.com>
parents: 261
diff changeset
1 import os.path
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import pytest
237
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
3 from piecrust.data.linker import Linker
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 from .mockutil import mock_fs, mock_fs_scope
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7 @pytest.mark.parametrize(
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
8 'fs_fac, page_path, expected',
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 [
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
10 (lambda: mock_fs().withPage('pages/foo'), 'foo.md',
237
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
11 # is_dir, name, is_self, data
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
12 [(False, 'foo', True, '/foo')]),
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
13 ((lambda: mock_fs()
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 .withPage('pages/foo')
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 .withPage('pages/bar')),
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16 'foo.md',
237
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
17 [(False, 'bar', False, '/bar'), (False, 'foo', True, '/foo')]),
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
18 ((lambda: mock_fs()
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19 .withPage('pages/baz')
237
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
20 .withPage('pages/something')
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 .withPage('pages/something/else')
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22 .withPage('pages/foo')
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 .withPage('pages/bar')),
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 'foo.md',
237
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
25 [(False, 'bar', False, '/bar'),
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
26 (False, 'baz', False, '/baz'),
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
27 (False, 'foo', True, '/foo'),
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
28 (True, 'something', False, '/something')]),
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
29 ((lambda: mock_fs()
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 .withPage('pages/something/else')
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 .withPage('pages/foo')
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 .withPage('pages/something/good')
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 .withPage('pages/bar')),
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 'something/else.md',
237
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
35 [(False, 'else', True, '/something/else'),
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
36 (False, 'good', False, '/something/good')])
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 ])
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
38 def test_linker_iteration(fs_fac, page_path, expected):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
39 fs = fs_fac()
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 with mock_fs_scope(fs):
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 app = fs.getApp()
261
b51ddb0c260b tests: Fix linker tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 237
diff changeset
42 app.config.set('site/pretty_urls', True)
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 src = app.getSource('pages')
404
27b10024f8d8 linker: Add ability to return the parent and ancestors of a page.
Ludovic Chabant <ludovic@chabant.com>
parents: 261
diff changeset
44 linker = Linker(src, os.path.dirname(page_path),
27b10024f8d8 linker: Add ability to return the parent and ancestors of a page.
Ludovic Chabant <ludovic@chabant.com>
parents: 261
diff changeset
45 root_page_path=page_path)
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 actual = list(iter(linker))
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 assert len(actual) == len(expected)
237
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
49 for (a, e) in zip(actual, expected):
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
50 is_dir, name, is_self, url = e
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
51 assert a.is_dir == is_dir
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
52 assert a.name == name
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
53 assert a.is_self == is_self
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
54 assert a.url == url
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 @pytest.mark.parametrize(
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
58 'fs_fac, page_path, expected',
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 [
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
60 (lambda: mock_fs().withPage('pages/foo'), 'foo.md',
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61 [('/foo', True)]),
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
62 ((lambda: mock_fs()
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63 .withPage('pages/foo')
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 .withPage('pages/bar')),
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 'foo.md',
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66 [('/bar', False), ('/foo', True)]),
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
67 ((lambda: mock_fs()
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68 .withPage('pages/baz')
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 .withPage('pages/something/else')
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70 .withPage('pages/foo')
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 .withPage('pages/bar')),
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 'foo.md',
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 [('/bar', False), ('/baz', False),
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 ('/foo', True), ('/something/else', False)]),
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
75 ((lambda: mock_fs()
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 .withPage('pages/something/else')
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 .withPage('pages/foo')
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 .withPage('pages/something/good')
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 .withPage('pages/bar')),
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 'something/else.md',
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81 [('/something/else', True),
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82 ('/something/good', False)])
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83 ])
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
84 def test_recursive_linker_iteration(fs_fac, page_path, expected):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 404
diff changeset
85 fs = fs_fac()
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
86 with mock_fs_scope(fs):
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
87 app = fs.getApp()
261
b51ddb0c260b tests: Fix linker tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 237
diff changeset
88 app.config.set('site/pretty_urls', True)
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
89 src = app.getSource('pages')
404
27b10024f8d8 linker: Add ability to return the parent and ancestors of a page.
Ludovic Chabant <ludovic@chabant.com>
parents: 261
diff changeset
90 linker = Linker(src, os.path.dirname(page_path),
27b10024f8d8 linker: Add ability to return the parent and ancestors of a page.
Ludovic Chabant <ludovic@chabant.com>
parents: 261
diff changeset
91 root_page_path=page_path)
237
879fe1457e48 data: `Linker` refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 212
diff changeset
92 actual = list(iter(linker.allpages))
212
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
93
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
94 assert len(actual) == len(expected)
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
95 for i, (a, e) in enumerate(zip(actual, expected)):
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
96 assert a.is_dir is False
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
97 assert a.url == e[0]
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98 assert a.is_self == e[1]
701591ebfcba data: Improve the Linker and RecursiveLinker features. Add tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
99