annotate tests/test_sources_posts.py @ 411:e7b865f8f335

bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.
author Ludovic Chabant <ludovic@chabant.com>
date Fri, 12 Jun 2015 17:09:19 -0700
parents cb6eadea0845
children 45ad976712ec
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
95
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import pytest
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 from .mockutil import mock_fs, mock_fs_scope
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
5 @pytest.mark.parametrize('fs_fac, src_type, expected_paths, expected_metadata', [
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
6 (lambda: mock_fs(), 'flat', [], []),
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
7 (lambda: mock_fs().withPage('test/2014-01-01_foo.md'),
95
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 'flat',
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 ['2014-01-01_foo.md'],
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10 [(2014, 1, 1, 'foo')]),
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
11 (lambda: mock_fs(), 'shallow', [], []),
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
12 (lambda: mock_fs().withPage('test/2014/01-01_foo.md'),
95
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13 'shallow',
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 ['2014/01-01_foo.md'],
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 [(2014, 1, 1, 'foo')]),
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
16 (lambda: mock_fs(), 'hierarchy', [], []),
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
17 (lambda: mock_fs().withPage('test/2014/01/01_foo.md'),
95
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 'hierarchy',
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19 ['2014/01/01_foo.md'],
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 [(2014, 1, 1, 'foo')]),
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 ])
411
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
22 def test_post_source_factories(fs_fac, src_type, expected_paths,
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
23 expected_metadata):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
24 fs = fs_fac()
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
25 fs.withConfig({
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
26 'site': {
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
27 'sources': {
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
28 'test': {'type': 'posts/%s' % src_type}},
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
29 'routes': [
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
30 {'url': '/%slug%', 'source': 'test'}]
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
31 }
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
32 })
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
33 fs.withDir('kitchen/test')
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
34 with mock_fs_scope(fs):
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
35 app = fs.getApp(cache=False)
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
36 s = app.getSource('test')
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
37 facs = list(s.buildPageFactories())
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
38 paths = [f.rel_path for f in facs]
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
39 assert paths == expected_paths
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
40 metadata = [
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
41 (f.metadata['year'], f.metadata['month'],
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
42 f.metadata['day'], f.metadata['slug'])
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
43 for f in facs]
e7b865f8f335 bake: Enable multiprocess baking.
Ludovic Chabant <ludovic@chabant.com>
parents: 95
diff changeset
44 assert metadata == expected_metadata
95
cb6eadea0845 Fixed a bug with the `shallow` source. Add unit tests.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45