piecrust2: piecrust/pipelines/_pagebaker.py annotate

annotate piecrust/pipelines/_pagebaker.py @ 979:45ad976712ec

tests: Big push to get the tests to pass again. - Lots of fixes everywhere in the code. - Try to handle debug logging in the multiprocessing worker pool when running in pytest. Not perfect, but usable for now. - Replace all `.md` test files with `.html` since now a auto-format extension always sets the format. - Replace `out` with `outfiles` in most places since now blog archives are added to the bake output and I don't want to add expected outputs for blog archives everywhere.

author	Ludovic Chabant <ludovic@chabant.com>
date	Sun, 29 Oct 2017 22:51:57 -0700
parents	8419daaa7a0e
children	8adc27285d93

rev	line source
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import os.path
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import queue
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	3 import shutil
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import logging
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import threading
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 import urllib.parse
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	7 from piecrust.pipelines._pagerecords import SubPagePipelineRecordEntry
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	8 from piecrust.rendering import RenderingContext, render_page
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	9 from piecrust.sources.base import AbortedSourceUseError
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	10 from piecrust.uriutil import split_uri
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	11
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13 logger = logging.getLogger(__name__)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	14
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	15
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	16 def get_output_path(app, out_dir, uri, pretty_urls):
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	17 uri_root, uri_path = split_uri(app, uri)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	18
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	19 bake_path = [out_dir]
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	20 decoded_uri = urllib.parse.unquote(uri_path)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	21 if pretty_urls:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	22 bake_path.append(decoded_uri)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	23 bake_path.append('index.html')
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	24 elif decoded_uri == '':
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	25 bake_path.append('index.html')
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	26 else:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	27 bake_path.append(decoded_uri)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	28
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	29 return os.path.normpath(os.path.join(*bake_path))
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	30
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	31
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32 class BakingError(Exception):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 pass
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36 class PageBaker(object):
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	37 def __init__(self, app, out_dir, force=False):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	38 self.app = app
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	39 self.out_dir = out_dir
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	40 self.force = force
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41 self.site_root = app.config.get('site/root')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42 self.pretty_urls = app.config.get('site/pretty_urls')
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	43 self._do_write = self._writeDirect
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44 self._writer_queue = None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	45 self._writer = None
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	46 self._stats = app.env.stats
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	47 self._rsr = app.env.rendered_segments_repository
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	48
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 def startWriterQueue(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50 self._writer_queue = queue.Queue()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51 self._writer = threading.Thread(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52 name='PageSerializer',
973 8419daaa7a0e internal: Make the page serializer thread daemon. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	53 daemon=True,
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54 target=_text_writer,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	55 args=(self._writer_queue,))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	56 self._writer.start()
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	57 self._do_write = self._sendToWriterQueue
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 def stopWriterQueue(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	60 self._writer_queue.put_nowait(None)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	61 self._writer.join()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	63 def _sendToWriterQueue(self, out_path, content):
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	64 self._writer_queue.put_nowait((out_path, content))
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	65
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	66 def _writeDirect(self, out_path, content):
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	67 with open(out_path, 'w', encoding='utf8') as fp:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	68 fp.write(content)
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	69
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	70 def bake(self, page, prev_entry, cur_entry):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71 cur_sub = 1
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	72 has_more_subs = True
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	73 app = self.app
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	74 out_dir = self.out_dir
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	75 pretty_urls = page.config.get('pretty_urls', self.pretty_urls)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	77 # Start baking the sub-pages.
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	78 while has_more_subs:
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	79 sub_uri = page.getUri(sub_num=cur_sub)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80 logger.debug("Baking '%s' [%d]..." % (sub_uri, cur_sub))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	81
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	82 out_path = get_output_path(app, out_dir, sub_uri, pretty_urls)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	83
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84 # Create the sub-entry for the bake record.
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	85 cur_sub_entry = SubPagePipelineRecordEntry(sub_uri, out_path)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	86
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	87 # Find a corresponding sub-entry in the previous bake record.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	88 prev_sub_entry = None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	89 if prev_entry is not None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	90 try:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	91 prev_sub_entry = prev_entry.getSub(cur_sub)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	92 except IndexError:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	93 pass
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	94
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	95 # Figure out if we need to bake this page.
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	96 bake_status = _get_bake_status(page, out_path, self.force,
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	97 prev_sub_entry, cur_sub_entry)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	98
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	99 # If this page didn't bake because it's already up-to-date.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	100 # Keep trying for as many subs as we know this page has.
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	101 if bake_status == STATUS_CLEAN:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	102 cur_sub_entry.render_info = prev_sub_entry.copyRenderInfo()
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	103 cur_sub_entry.flags = SubPagePipelineRecordEntry.FLAG_NONE
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	104 cur_entry.subs.append(cur_sub_entry)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	105
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	106 if prev_entry.num_subs >= cur_sub + 1:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	107 cur_sub += 1
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	108 has_more_subs = True
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	109 logger.debug(" %s is up to date, skipping to next "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	110 "sub-page." % out_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	111 continue
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	112
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	113 logger.debug(" %s is up to date, skipping bake." % out_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	114 break
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	115
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	116 # All good, proceed.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	117 try:
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	118 if bake_status == STATUS_INVALIDATE_AND_BAKE:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	119 cache_key = sub_uri
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	120 self._rsr.invalidate(cache_key)
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	121 cur_sub_entry.flags \|= \
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	122 SubPagePipelineRecordEntry.FLAG_FORMATTING_INVALIDATED
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	123
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	124 logger.debug(" p%d -> %s" % (cur_sub, out_path))
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	125 rp = self._bakeSingle(page, cur_sub, out_path)
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	126 except AbortedSourceUseError:
d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	127 raise
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	128 except Exception as ex:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	129 logger.exception(ex)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	130 raise BakingError("%s: error baking '%s'." %
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	131 (page.content_spec, sub_uri)) from ex
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	132
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	133 # Record what we did.
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	134 cur_sub_entry.flags \|= SubPagePipelineRecordEntry.FLAG_BAKED
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	135 cur_sub_entry.render_info = rp.copyRenderInfo()
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	136 cur_entry.subs.append(cur_sub_entry)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	137
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	138 # Copy page assets.
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	139 if (cur_sub == 1 and
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	140 cur_sub_entry.anyPass(lambda p: p.used_assets)):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	141 if pretty_urls:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	142 out_assets_dir = os.path.dirname(out_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	143 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	144 out_assets_dir, out_name = os.path.split(out_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	145 if sub_uri != self.site_root:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	146 out_name_noext, _ = os.path.splitext(out_name)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	147 out_assets_dir = os.path.join(out_assets_dir,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	148 out_name_noext)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	149
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150 logger.debug("Copying page assets to: %s" % out_assets_dir)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	151 _ensure_dir_exists(out_assets_dir)
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	152 assetor = rp.data.get('assets')
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	153 if assetor is not None:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	154 for i in assetor._getAssetItems():
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	155 fn = os.path.basename(i.spec)
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	156 out_asset_path = os.path.join(out_assets_dir, fn)
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	157 logger.debug(" %s -> %s" % (i.spec, out_asset_path))
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	158 shutil.copy(i.spec, out_asset_path)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	159
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	160 # Figure out if we have more work.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	161 has_more_subs = False
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	162 if cur_sub_entry.anyPass(lambda p: p.pagination_has_more):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	163 cur_sub += 1
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	164 has_more_subs = True
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	165
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	166 def _bakeSingle(self, page, sub_num, out_path):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	167 ctx = RenderingContext(page, sub_num=sub_num)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	168 page.source.prepareRenderContext(ctx)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	169
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	170 with self._stats.timerScope("PageRender"):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	171 rp = render_page(ctx)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	172
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	173 with self._stats.timerScope("PageSerialize"):
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	174 self._do_write(out_path, rp.content)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	175
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	176 return rp
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	177
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	178
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	179 def _text_writer(q):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	180 while True:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	181 item = q.get()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	182 if item is not None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	183 out_path, txt = item
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	184 out_dir = os.path.dirname(out_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	185 _ensure_dir_exists(out_dir)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	186
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	187 with open(out_path, 'w', encoding='utf8') as fp:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	188 fp.write(txt)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	189
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	190 q.task_done()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	191 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	192 # Sentinel object, terminate the thread.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	193 q.task_done()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	194 break
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	195
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	196
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	197 STATUS_CLEAN = 0
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	198 STATUS_BAKE = 1
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	199 STATUS_INVALIDATE_AND_BAKE = 2
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	200
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	201
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	202 def _get_bake_status(page, out_path, force, prev_sub_entry, cur_sub_entry):
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	203 # Figure out if we need to invalidate or force anything.
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	204 status = _compute_force_flags(prev_sub_entry, cur_sub_entry)
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	205 if status != STATUS_CLEAN:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	206 return status
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	207
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	208 # Easy test.
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	209 if force:
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	210 cur_sub_entry.flags \|= \
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	211 SubPagePipelineRecordEntry.FLAG_FORCED_BY_GENERAL_FORCE
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	212 return STATUS_BAKE
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	213
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	214 # Check for up-to-date outputs.
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	215 in_path_time = page.content_mtime
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	216 try:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	217 out_path_time = os.path.getmtime(out_path)
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	218 except OSError:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	219 # File doesn't exist, we'll need to bake.
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	220 cur_sub_entry.flags \|= \
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	221 SubPagePipelineRecordEntry.FLAG_FORCED_BY_NO_PREVIOUS
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	222 return STATUS_BAKE
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	223
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	224 if out_path_time <= in_path_time:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	225 return STATUS_BAKE
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	226
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	227 # Nope, all good.
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	228 return STATUS_CLEAN
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	229
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	230
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	231 def _compute_force_flags(prev_sub_entry, cur_sub_entry):
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	232 if prev_sub_entry and prev_sub_entry.errors:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	233 # Previous bake failed. We'll have to bake it again.
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	234 cur_sub_entry.flags \|= \
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	235 SubPagePipelineRecordEntry.FLAG_FORCED_BY_PREVIOUS_ERRORS
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	236 return STATUS_BAKE
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	237
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	238 if not prev_sub_entry:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	239 cur_sub_entry.flags \|= \
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	240 SubPagePipelineRecordEntry.FLAG_FORCED_BY_NO_PREVIOUS
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	241 return STATUS_BAKE
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	242
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	243 return STATUS_CLEAN
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	244
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	245
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	246 def _ensure_dir_exists(path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	247 try:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	248 os.makedirs(path, mode=0o755, exist_ok=True)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	249 except OSError:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	250 # In a multiprocess environment, several process may very
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	251 # occasionally try to create the same directory at the same time.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	252 # Let's ignore any error and if something's really wrong (like file
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	253 # acces permissions or whatever), then it will more legitimately fail
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	254 # just after this when we try to write files.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	255 pass
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	256

Mercurial > piecrust2

annotate piecrust/pipelines/_pagebaker.py @ 979:45ad976712ec