Mercurial > piecrust2

annotate tests/test_data_assetor.py @ 411:e7b865f8f335

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.
author Ludovic Chabant <ludovic@chabant.com>
date Fri, 12 Jun 2015 17:09:19 -0700
parents 422052d2e978
children f987b29d6fab
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import pytest
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 from mock import MagicMock
329
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
3 from piecrust.data.assetor import (
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
4 Assetor, UnsupportedAssetsError, build_base_url)
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 from .mockutil import mock_fs, mock_fs_scope
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 329
diff changeset
8 @pytest.mark.parametrize('fs_fac, site_root, expected', [
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 329
diff changeset
9 (lambda: mock_fs().withPage('pages/foo/bar'), '/', {}),
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 329
diff changeset
10 (lambda: mock_fs()
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
11 .withPage('pages/foo/bar')
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
12 .withPageAsset('pages/foo/bar', 'one.txt', 'one'),
329
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
13 '/',
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 {'one': 'one'}),
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 329
diff changeset
15 (lambda: mock_fs()
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
16 .withPage('pages/foo/bar')
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
17 .withPageAsset('pages/foo/bar', 'one.txt', 'one')
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
18 .withPageAsset('pages/foo/bar', 'two.txt', 'two'),
329
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
19 '/',
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
20 {'one': 'one', 'two': 'two'}),
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
21
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 329
diff changeset
22 (lambda: mock_fs().withPage('pages/foo/bar'), '/whatever', {}),
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 329
diff changeset
23 (lambda: mock_fs()
329
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
24 .withPage('pages/foo/bar')
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
25 .withPageAsset('pages/foo/bar', 'one.txt', 'one'),
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
26 '/whatever',
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
27 {'one': 'one'}),
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 329
diff changeset
28 (lambda: mock_fs()
329
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
29 .withPage('pages/foo/bar')
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
30 .withPageAsset('pages/foo/bar', 'one.txt', 'one')
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
31 .withPageAsset('pages/foo/bar', 'two.txt', 'two'),
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
32 '/whatever',
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 {'one': 'one', 'two': 'two'})
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 ])
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 329
diff changeset
35 def test_assets(fs_fac, site_root, expected):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 329
diff changeset
36 fs = fs_fac()
329
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
37 fs.withConfig({'site': {'root': site_root}})
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 with mock_fs_scope(fs):
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 page = MagicMock()
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
40 page.app = fs.getApp(cache=False)
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
41 page.app.env.base_asset_url_format = '%uri%'
36
485682a6de50 New site layout support.
Ludovic Chabant <ludovic@chabant.com>
parents: 33
diff changeset
42 page.path = fs.path('/kitchen/pages/foo/bar.md')
329
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
43 assetor = Assetor(page, site_root.rstrip('/') + '/foo/bar')
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 for en in expected.keys():
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 assert hasattr(assetor, en)
329
422052d2e978 internal: Try handling URLs in a consistent way.
Ludovic Chabant <ludovic@chabant.com>
parents: 316
diff changeset
46 path = site_root.rstrip('/') + '/foo/bar/%s.txt' % en
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 assert getattr(assetor, en) == path
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 assert assetor[en] == path
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 def test_missing_asset():
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 with pytest.raises(KeyError):
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
53 fs = mock_fs().withPage('pages/foo/bar')
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 with mock_fs_scope(fs):
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 page = MagicMock()
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
56 page.app = fs.getApp(cache=False)
36
485682a6de50 New site layout support.
Ludovic Chabant <ludovic@chabant.com>
parents: 33
diff changeset
57 page.path = fs.path('/kitchen/pages/foo/bar.md')
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 assetor = Assetor(page, '/foo/bar')
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 assetor['this_doesnt_exist']
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62 def test_multiple_assets_with_same_name():
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63 with pytest.raises(UnsupportedAssetsError):
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 fs = (mock_fs()
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
65 .withPage('pages/foo/bar')
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
66 .withPageAsset('pages/foo/bar', 'one.txt', 'one text')
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
67 .withPageAsset('pages/foo/bar', 'one.jpg', 'one picture'))
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68 with mock_fs_scope(fs):
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 page = MagicMock()
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 36
diff changeset
70 page.app = fs.getApp(cache=False)
36
485682a6de50 New site layout support.
Ludovic Chabant <ludovic@chabant.com>
parents: 33
diff changeset
71 page.path = fs.path('/kitchen/pages/foo/bar.md')
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 assetor = Assetor(page, '/foo/bar')
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 assetor['one']
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
75
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
76 @pytest.mark.parametrize('url_format, pretty_urls, uri, expected', [
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
77 ('%uri%', True, '/foo', '/foo/'),
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
78 ('%uri%', True, '/foo.ext', '/foo.ext/'),
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
79 ('%uri%', False, '/foo.html', '/foo/'),
33
62c7a97c8340 Get the un-paginated URL of a page early and pass that around.
Ludovic Chabant <ludovic@chabant.com>
parents: 32
diff changeset
80 ('%uri%', False, '/foo.ext', '/foo/'),
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
81 ])
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
82 def test_build_base_url(url_format, pretty_urls, uri, expected):
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
83 app = MagicMock()
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
84 app.env = MagicMock()
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
85 app.env.base_asset_url_format = url_format
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
86 app.config = {
316
eb958151c8dc tests: Bad me, the tests were broken. Now they're fixed.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
87 'site/root': '/',
33
62c7a97c8340 Get the un-paginated URL of a page early and pass that around.
Ludovic Chabant <ludovic@chabant.com>
parents: 32
diff changeset
88 'site/pretty_urls': pretty_urls}
32
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
89 assets_path = 'foo/bar-assets'
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
90 actual = build_base_url(app, uri, assets_path)
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
91 assert actual == expected
43091c9837bf Fix problems with asset URLs.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
92