annotate tests/test_baking_baker.py @ 215:a47580a0955b

bake: Better error handling for the processing pipeline. Pipeline jobs now keep track of whether they've seen any errors. This is aggregated into an overall "success" flag for the processing record. Also, jobs keep going as long as there's no critical (i.e. internal) failure happening. Errors raised by processors are also better tracked: the actual processor that failed, along with the input file, are tracks in the processing record. The `bake` command returns a failure exit code if processing saw any error.
author Ludovic Chabant <ludovic@chabant.com>
date Sat, 31 Jan 2015 17:08:02 -0800
parents dce37d1d4f05
children 61145dcd56e0
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
11
617191dec18e Fixes for Windows, make `findPagePath` return a ref path.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
1 import os.path
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import pytest
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
3 from piecrust.baking.baker import PageBaker, Baker
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
4 from piecrust.baking.records import BakeRecord
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
5 from .mockutil import get_mock_app, mock_fs, mock_fs_scope
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 @pytest.mark.parametrize('uri, page_num, pretty, expected', [
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 # Pretty URLs
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10 ('', 1, True, 'index.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11 ('', 2, True, '2/index.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12 ('foo', 1, True, 'foo/index.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13 ('foo', 2, True, 'foo/2/index.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 ('foo/bar', 1, True, 'foo/bar/index.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 ('foo/bar', 2, True, 'foo/bar/2/index.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16 ('foo.ext', 1, True, 'foo.ext/index.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 ('foo.ext', 2, True, 'foo.ext/2/index.html'),
29
7e44f6092a1d More unit tests for output bake paths.
Ludovic Chabant <ludovic@chabant.com>
parents: 11
diff changeset
18 ('foo/bar.ext', 1, True, 'foo/bar.ext/index.html'),
7e44f6092a1d More unit tests for output bake paths.
Ludovic Chabant <ludovic@chabant.com>
parents: 11
diff changeset
19 ('foo/bar.ext', 2, True, 'foo/bar.ext/2/index.html'),
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 ('foo.bar.ext', 1, True, 'foo.bar.ext/index.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 ('foo.bar.ext', 2, True, 'foo.bar.ext/2/index.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22 # Ugly URLs
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 ('', 1, False, 'index.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 ('', 2, False, '2.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 ('foo', 1, False, 'foo.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 ('foo', 2, False, 'foo/2.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 ('foo/bar', 1, False, 'foo/bar.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 ('foo/bar', 2, False, 'foo/bar/2.html'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29 ('foo.ext', 1, False, 'foo.ext'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 ('foo.ext', 2, False, 'foo/2.ext'),
29
7e44f6092a1d More unit tests for output bake paths.
Ludovic Chabant <ludovic@chabant.com>
parents: 11
diff changeset
31 ('foo/bar.ext', 1, False, 'foo/bar.ext'),
7e44f6092a1d More unit tests for output bake paths.
Ludovic Chabant <ludovic@chabant.com>
parents: 11
diff changeset
32 ('foo/bar.ext', 2, False, 'foo/bar/2.ext'),
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 ('foo.bar.ext', 1, False, 'foo.bar.ext'),
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 ('foo.bar.ext', 2, False, 'foo.bar/2.ext')
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35 ])
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 def test_get_output_path(uri, page_num, pretty, expected):
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 app = get_mock_app()
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 if pretty:
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 app.config.set('site/pretty_urls', True)
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 assert app.config.get('site/pretty_urls') == pretty
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 baker = PageBaker(app, '/destination')
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 sub_uri = baker.getOutputUri(uri, page_num)
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 path = baker.getOutputPath(sub_uri)
11
617191dec18e Fixes for Windows, make `findPagePath` return a ref path.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
45 expected = os.path.normpath(
617191dec18e Fixes for Windows, make `findPagePath` return a ref path.
Ludovic Chabant <ludovic@chabant.com>
parents: 6
diff changeset
46 os.path.join('/destination', expected))
6
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 assert expected == path
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
49
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
50 def test_empty_bake():
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
51 fs = mock_fs()
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
52 with mock_fs_scope(fs):
145
dce37d1d4f05 Fix unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
53 out_dir = fs.path('kitchen/_counter')
dce37d1d4f05 Fix unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
54 assert not os.path.isdir(out_dir)
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
55 app = fs.getApp()
145
dce37d1d4f05 Fix unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
56 baker = Baker(app, out_dir)
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
57 baker.bake()
145
dce37d1d4f05 Fix unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
58 assert os.path.isdir(out_dir)
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
59 structure = fs.getStructure('kitchen/_counter')
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
60 assert list(structure.keys()) == ['index.html']
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
61
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
62
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
63 def test_simple_bake():
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
64 fs = (mock_fs()
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
65 .withPage('posts/2010-01-01_post1.md', {'layout': 'none', 'format': 'none'}, 'post one')
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
66 .withPage('pages/_index.md', {'layout': 'none', 'format': 'none'}, "something"))
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
67 with mock_fs_scope(fs):
145
dce37d1d4f05 Fix unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
68 out_dir = fs.path('kitchen/_counter')
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
69 app = fs.getApp()
145
dce37d1d4f05 Fix unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
70 baker = Baker(app, out_dir)
85
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
71 baker.bake()
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
72 structure = fs.getStructure('kitchen/_counter')
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
73 assert structure == {
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
74 '2010': {'01': {'01': {'post1.html': 'post one'}}},
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
75 'index.html': 'something'}
3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 29
diff changeset
76
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
77 def test_removed():
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
78 fs = (mock_fs()
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
79 .withPage('pages/foo.md', {'layout': 'none', 'format': 'none'}, 'a foo page')
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
80 .withPage('pages/_index.md', {'layout': 'none', 'format': 'none'}, "something"))
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
81 with mock_fs_scope(fs):
145
dce37d1d4f05 Fix unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
82 out_dir = fs.path('kitchen/_counter')
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
83 app = fs.getApp()
145
dce37d1d4f05 Fix unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
84 baker = Baker(app, out_dir)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
85 baker.bake()
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
86 structure = fs.getStructure('kitchen/_counter')
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
87 assert structure == {
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
88 'foo.html': 'a foo page',
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
89 'index.html': 'something'}
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
90
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
91 os.remove(fs.path('kitchen/pages/foo.md'))
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
92 app = fs.getApp()
145
dce37d1d4f05 Fix unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
93 baker = Baker(app, out_dir)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
94 baker.bake()
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
95 structure = fs.getStructure('kitchen/_counter')
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
96 assert structure == {
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
97 'index.html': 'something'}
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
98
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
99 def test_record_version_change():
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
100 fs = (mock_fs()
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
101 .withPage('pages/foo.md', {'layout': 'none', 'format': 'none'}, 'a foo page'))
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
102 with mock_fs_scope(fs):
145
dce37d1d4f05 Fix unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
103 out_dir = fs.path('kitchen/_counter')
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
104 app = fs.getApp()
145
dce37d1d4f05 Fix unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
105 baker = Baker(app, out_dir)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
106 baker.bake()
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
107 mtime = os.path.getmtime(fs.path('kitchen/_counter/foo.html'))
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
108
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
109 app = fs.getApp()
145
dce37d1d4f05 Fix unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
110 baker = Baker(app, out_dir)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
111 baker.bake()
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
112 assert mtime == os.path.getmtime(fs.path('kitchen/_counter/foo.html'))
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
113
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
114 BakeRecord.RECORD_VERSION += 1
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
115 try:
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
116 app = fs.getApp()
145
dce37d1d4f05 Fix unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents: 120
diff changeset
117 baker = Baker(app, out_dir)
120
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
118 baker.bake()
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
119 assert mtime < os.path.getmtime(fs.path('kitchen/_counter/foo.html'))
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
120 finally:
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
121 BakeRecord.RECORD_VERSION -= 1
133845647083 Better error management and removal support in baking/processing.
Ludovic Chabant <ludovic@chabant.com>
parents: 85
diff changeset
122