piecrust2: tests/test_baking

annotate tests/test_baking_baker.py @ 411:e7b865f8f335

bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.

author	Ludovic Chabant <ludovic@chabant.com>
date	Fri, 12 Jun 2015 17:09:19 -0700
parents	76c838453dbe
children	6b6c5442c790

rev	line source
286 a2d283d1033d tests: Fixes for running on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 262 diff changeset	1 import time
11 617191dec18e Fixes for Windows, make `findPagePath` return a ref path. Ludovic Chabant <ludovic@chabant.com> parents: 6 diff changeset	2 import os.path
6 f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import pytest
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 347 diff changeset	4 from piecrust.baking.baker import Baker
e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 347 diff changeset	5 from piecrust.baking.single import PageBaker
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	6 from piecrust.baking.records import BakeRecord
85 3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests. Ludovic Chabant <ludovic@chabant.com> parents: 29 diff changeset	7 from .mockutil import get_mock_app, mock_fs, mock_fs_scope
6 f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	9
262 61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	10 @pytest.mark.parametrize('uri, pretty, expected', [
6 f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	11 # Pretty URLs
262 61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	12 ('', True, 'index.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	13 ('2', True, '2/index.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	14 ('foo', True, 'foo/index.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	15 ('foo/2', True, 'foo/2/index.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	16 ('foo/bar', True, 'foo/bar/index.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	17 ('foo/bar/2', True, 'foo/bar/2/index.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	18 ('foo.ext', True, 'foo.ext/index.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	19 ('foo.ext/2', True, 'foo.ext/2/index.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	20 ('foo/bar.ext', True, 'foo/bar.ext/index.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	21 ('foo/bar.ext/2', True, 'foo/bar.ext/2/index.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	22 ('foo.bar.ext', True, 'foo.bar.ext/index.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	23 ('foo.bar.ext/2', True, 'foo.bar.ext/2/index.html'),
6 f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24 # Ugly URLs
262 61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	25 ('', False, 'index.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	26 ('2.html', False, '2.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	27 ('foo.html', False, 'foo.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	28 ('foo/2.html', False, 'foo/2.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	29 ('foo/bar.html', False, 'foo/bar.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	30 ('foo/bar/2.html', False, 'foo/bar/2.html'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	31 ('foo.ext', False, 'foo.ext'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	32 ('foo/2.ext', False, 'foo/2.ext'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	33 ('foo/bar.ext', False, 'foo/bar.ext'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	34 ('foo/bar/2.ext', False, 'foo/bar/2.ext'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	35 ('foo.bar.ext', False, 'foo.bar.ext'),
61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	36 ('foo.bar/2.ext', False, 'foo.bar/2.ext')
6 f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37 ])
262 61145dcd56e0 routing: Better generate URLs according to the site configuration. Ludovic Chabant <ludovic@chabant.com> parents: 145 diff changeset	38 def test_get_output_path(uri, pretty, expected):
6 f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	39 app = get_mock_app()
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	40 if pretty:
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41 app.config.set('site/pretty_urls', True)
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42 assert app.config.get('site/pretty_urls') == pretty
f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	43
329 422052d2e978 internal: Try handling URLs in a consistent way. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	44 for site_root in ['/', '/whatever/']:
422052d2e978 internal: Try handling URLs in a consistent way. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	45 app.config.set('site/root', site_root)
422052d2e978 internal: Try handling URLs in a consistent way. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	46 baker = PageBaker(app, '/destination')
422052d2e978 internal: Try handling URLs in a consistent way. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	47 path = baker.getOutputPath(site_root + uri)
422052d2e978 internal: Try handling URLs in a consistent way. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	48 expected = os.path.normpath(
422052d2e978 internal: Try handling URLs in a consistent way. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	49 os.path.join('/destination', expected))
422052d2e978 internal: Try handling URLs in a consistent way. Ludovic Chabant <ludovic@chabant.com> parents: 324 diff changeset	50 assert expected == path
6 f5ca5c5bed85 More Python 3 fixes, modularization, and new unit tests. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51
85 3471ffa059b2 Add a `BakeScheduler` to handle build dependencies. Add unit-tests. Ludovic Chabant <ludovic@chabant.com> parents: 29 diff changeset	52
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	53 def test_removed():
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	54 fs = (mock_fs()
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	55 .withPage('pages/foo.md', {'layout': 'none', 'format': 'none'}, 'a foo page')
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	56 .withPage('pages/_index.md', {'layout': 'none', 'format': 'none'}, "something"))
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	57 with mock_fs_scope(fs):
145 dce37d1d4f05 Fix unit tests. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	58 out_dir = fs.path('kitchen/_counter')
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	59 app = fs.getApp()
411 e7b865f8f335 bake: Enable multiprocess baking. Ludovic Chabant <ludovic@chabant.com> parents: 347 diff changeset	60 app.config.set('baker/workers', 1)
145 dce37d1d4f05 Fix unit tests. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	61 baker = Baker(app, out_dir)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	62 baker.bake()
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	63 structure = fs.getStructure('kitchen/_counter')
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	64 assert structure == {
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	65 'foo.html': 'a foo page',
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	66 'index.html': 'something'}
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	67
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	68 os.remove(fs.path('kitchen/pages/foo.md'))
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	69 app = fs.getApp()
145 dce37d1d4f05 Fix unit tests. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	70 baker = Baker(app, out_dir)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	71 baker.bake()
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	72 structure = fs.getStructure('kitchen/_counter')
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	73 assert structure == {
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	74 'index.html': 'something'}
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	75
324 65e6d72f3877 bake/serve: Fix how taxonomy index pages are setup and rendered. Ludovic Chabant <ludovic@chabant.com> parents: 286 diff changeset	76
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	77 def test_record_version_change():
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	78 fs = (mock_fs()
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	79 .withPage('pages/foo.md', {'layout': 'none', 'format': 'none'}, 'a foo page'))
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	80 with mock_fs_scope(fs):
145 dce37d1d4f05 Fix unit tests. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	81 out_dir = fs.path('kitchen/_counter')
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	82 app = fs.getApp()
145 dce37d1d4f05 Fix unit tests. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	83 baker = Baker(app, out_dir)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	84 baker.bake()
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	85 mtime = os.path.getmtime(fs.path('kitchen/_counter/foo.html'))
286 a2d283d1033d tests: Fixes for running on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 262 diff changeset	86 time.sleep(1)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	87
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	88 app = fs.getApp()
145 dce37d1d4f05 Fix unit tests. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	89 baker = Baker(app, out_dir)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	90 baker.bake()
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	91 assert mtime == os.path.getmtime(fs.path('kitchen/_counter/foo.html'))
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	92
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	93 BakeRecord.RECORD_VERSION += 1
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	94 try:
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	95 app = fs.getApp()
145 dce37d1d4f05 Fix unit tests. Ludovic Chabant <ludovic@chabant.com> parents: 120 diff changeset	96 baker = Baker(app, out_dir)
120 133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	97 baker.bake()
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	98 assert mtime < os.path.getmtime(fs.path('kitchen/_counter/foo.html'))
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	99 finally:
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	100 BakeRecord.RECORD_VERSION -= 1
133845647083 Better error management and removal support in baking/processing. Ludovic Chabant <ludovic@chabant.com> parents: 85 diff changeset	101

Mercurial > piecrust2

annotate tests/test_baking_baker.py @ 411:e7b865f8f335