piecrust2: piecrust/pipelines/_pagebaker.py annotate

annotate piecrust/pipelines/_pagebaker.py @ 1051:971b4d67e82a

serve: Fix problems with assets disappearing between servings. When an asset file changes, its source's pipeline is re-run. But that created a bake record that only had that pipeline's output, so the other outputs were incorrectly considered empty and therefore any stray files were removed. Now we copy over bake records for the pipelines we don't run.

author	Ludovic Chabant <ludovic@chabant.com>
date	Fri, 26 Jan 2018 18:05:02 -0800
parents	298b07a899b5
children	3bcb2d446397

rev	line source
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import os.path
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	2 import copy
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import queue
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	4 import shutil
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import logging
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 import threading
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	7 import urllib.parse
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	8 from piecrust.pipelines._pagerecords import (
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	9 SubPageFlags, create_subpage_job_result)
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	10 from piecrust.rendering import RenderingContext, render_page
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	11 from piecrust.sources.base import AbortedSourceUseError
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12 from piecrust.uriutil import split_uri
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	14
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	15 logger = logging.getLogger(__name__)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	18 def get_output_path(app, out_dir, uri, pretty_urls):
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	19 uri_root, uri_path = split_uri(app, uri)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	20
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	21 bake_path = [out_dir]
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	22 decoded_uri = urllib.parse.unquote(uri_path)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	23 if pretty_urls:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	24 bake_path.append(decoded_uri)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	25 bake_path.append('index.html')
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	26 elif decoded_uri == '':
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	27 bake_path.append('index.html')
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	28 else:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	29 bake_path.append(decoded_uri)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	30
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	31 return os.path.normpath(os.path.join(*bake_path))
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	32
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	33
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34 class BakingError(Exception):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	35 pass
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	37
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	38 class PageBaker(object):
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	39 def __init__(self, app, out_dir, force=False):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	40 self.app = app
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	41 self.out_dir = out_dir
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42 self.force = force
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	43 self.site_root = app.config.get('site/root')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44 self.pretty_urls = app.config.get('site/pretty_urls')
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	45 self._do_write = self._writeDirect
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	46 self._writer_queue = None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	47 self._writer = None
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	48 self._stats = app.env.stats
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	49 self._rsr = app.env.rendered_segments_repository
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51 def startWriterQueue(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52 self._writer_queue = queue.Queue()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53 self._writer = threading.Thread(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54 name='PageSerializer',
973 8419daaa7a0e internal: Make the page serializer thread daemon. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	55 daemon=True,
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	56 target=_text_writer,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	57 args=(self._writer_queue,))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58 self._writer.start()
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	59 self._do_write = self._sendToWriterQueue
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	60
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	61 def stopWriterQueue(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62 self._writer_queue.put_nowait(None)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63 self._writer.join()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	65 def _sendToWriterQueue(self, out_path, content):
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	66 self._writer_queue.put_nowait((out_path, content))
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	67
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	68 def _writeDirect(self, out_path, content):
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	69 with open(out_path, 'w', encoding='utf8') as fp:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	70 fp.write(content)
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	71
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	72 def bake(self, page, prev_entry, force=False):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	73 cur_sub = 1
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	74 has_more_subs = True
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	75 app = self.app
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	76 out_dir = self.out_dir
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	77 force_bake = self.force or force
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	78 pretty_urls = page.config.get('pretty_urls', self.pretty_urls)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	79
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	80 rendered_subs = []
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	81
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	82 # Start baking the sub-pages.
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	83 while has_more_subs:
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	84 sub_uri = page.getUri(sub_num=cur_sub)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	85 logger.debug("Baking '%s' [%d]..." % (sub_uri, cur_sub))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	86
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 973 diff changeset	87 out_path = get_output_path(app, out_dir, sub_uri, pretty_urls)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	88
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	89 # Create the sub-entry for the bake record.
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	90 cur_sub_entry = create_subpage_job_result(sub_uri, out_path)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	91 rendered_subs.append(cur_sub_entry)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	92
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	93 # Find a corresponding sub-entry in the previous bake record.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	94 prev_sub_entry = None
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	95 if prev_entry is not None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	96 try:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	97 prev_sub_entry = prev_entry.getSub(cur_sub)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	98 except IndexError:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	99 pass
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	100
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	101 # Figure out if we need to bake this page.
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	102 bake_status = _get_bake_status(page, out_path, force_bake,
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	103 prev_sub_entry, cur_sub_entry)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	104
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	105 # If this page didn't bake because it's already up-to-date.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	106 # Keep trying for as many subs as we know this page has.
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	107 if bake_status == STATUS_CLEAN:
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	108 cur_sub_entry['render_info'] = copy.deepcopy(
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	109 prev_sub_entry['render_info'])
1020 298b07a899b5 bake: Fix overriding issues between theme and user pages for index pages. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	110 cur_sub_entry['flags'] = \
298b07a899b5 bake: Fix overriding issues between theme and user pages for index pages. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	111 SubPageFlags.FLAG_COLLAPSED_FROM_LAST_RUN
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	112
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	113 if prev_entry.num_subs >= cur_sub + 1:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	114 cur_sub += 1
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	115 has_more_subs = True
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	116 logger.debug(" %s is up to date, skipping to next "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	117 "sub-page." % out_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	118 continue
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	119
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	120 logger.debug(" %s is up to date, skipping bake." % out_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	121 break
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	122
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	123 # All good, proceed.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	124 try:
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	125 if bake_status == STATUS_INVALIDATE_AND_BAKE:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	126 cache_key = sub_uri
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	127 self._rsr.invalidate(cache_key)
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	128 cur_sub_entry['flags'] \|= \
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	129 SubPageFlags.FLAG_RENDER_CACHE_INVALIDATED
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	130
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	131 logger.debug(" p%d -> %s" % (cur_sub, out_path))
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	132 rp = self._bakeSingle(page, cur_sub, out_path)
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	133 except AbortedSourceUseError:
d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 871 diff changeset	134 raise
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	135 except Exception as ex:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	136 logger.exception(ex)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	137 raise BakingError("%s: error baking '%s'." %
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	138 (page.content_spec, sub_uri)) from ex
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	139
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	140 # Record what we did.
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	141 cur_sub_entry['flags'] \|= SubPageFlags.FLAG_BAKED
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	142 cur_sub_entry['render_info'] = copy.deepcopy(rp.render_info)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	143
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	144 # Copy page assets.
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	145 if (cur_sub == 1 and
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	146 cur_sub_entry['render_info']['used_assets']):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	147 if pretty_urls:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	148 out_assets_dir = os.path.dirname(out_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	149 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150 out_assets_dir, out_name = os.path.split(out_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	151 if sub_uri != self.site_root:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	152 out_name_noext, _ = os.path.splitext(out_name)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	153 out_assets_dir = os.path.join(out_assets_dir,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	154 out_name_noext)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	155
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	156 logger.debug("Copying page assets to: %s" % out_assets_dir)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	157 _ensure_dir_exists(out_assets_dir)
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	158 assetor = rp.data.get('assets')
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	159 if assetor is not None:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	160 for i in assetor._getAssetItems():
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	161 fn = os.path.basename(i.spec)
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	162 out_asset_path = os.path.join(out_assets_dir, fn)
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	163 logger.debug(" %s -> %s" % (i.spec, out_asset_path))
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	164 shutil.copy(i.spec, out_asset_path)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	165
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	166 # Figure out if we have more work.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	167 has_more_subs = False
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	168 if cur_sub_entry['render_info']['pagination_has_more']:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	169 cur_sub += 1
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	170 has_more_subs = True
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	171
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	172 return rendered_subs
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	173
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	174 def _bakeSingle(self, page, sub_num, out_path):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	175 ctx = RenderingContext(page, sub_num=sub_num)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	176 page.source.prepareRenderContext(ctx)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	177
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	178 with self._stats.timerScope("PageRender"):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	179 rp = render_page(ctx)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	180
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	181 with self._stats.timerScope("PageSerialize"):
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	182 self._do_write(out_path, rp.content)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	183
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	184 return rp
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	185
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	186
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	187 def _text_writer(q):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	188 while True:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	189 item = q.get()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	190 if item is not None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	191 out_path, txt = item
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	192 out_dir = os.path.dirname(out_path)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	193 _ensure_dir_exists(out_dir)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	194
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	195 with open(out_path, 'w', encoding='utf8') as fp:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	196 fp.write(txt)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	197
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	198 q.task_done()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	199 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	200 # Sentinel object, terminate the thread.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	201 q.task_done()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	202 break
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	203
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	204
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	205 STATUS_CLEAN = 0
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	206 STATUS_BAKE = 1
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	207 STATUS_INVALIDATE_AND_BAKE = 2
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	208
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	209
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	210 def _get_bake_status(page, out_path, force, prev_sub_entry, cur_sub_entry):
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	211 # Figure out if we need to invalidate or force anything.
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	212 status = _compute_force_flags(prev_sub_entry, cur_sub_entry)
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	213 if status != STATUS_CLEAN:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	214 return status
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	215
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	216 # Easy test.
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	217 if force:
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	218 cur_sub_entry['flags'] \|= \
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	219 SubPageFlags.FLAG_FORCED_BY_GENERAL_FORCE
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	220 # We need to invalidate any cache we have on this page because
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	221 # it's being forced, so something important has changed somehow.
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	222 return STATUS_INVALIDATE_AND_BAKE
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	223
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	224 # Check for up-to-date outputs.
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	225 in_path_time = page.content_mtime
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	226 try:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	227 out_path_time = os.path.getmtime(out_path)
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	228 except OSError:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	229 # File doesn't exist, we'll need to bake.
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	230 cur_sub_entry['flags'] \|= \
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	231 SubPageFlags.FLAG_FORCED_BY_NO_PREVIOUS
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	232 return STATUS_BAKE
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	233
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	234 if out_path_time <= in_path_time:
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	235 return STATUS_BAKE
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	236
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	237 # Nope, all good.
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	238 return STATUS_CLEAN
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	239
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	240
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	241 def _compute_force_flags(prev_sub_entry, cur_sub_entry):
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	242 if prev_sub_entry and len(prev_sub_entry['errors']) > 0:
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	243 # Previous bake failed. We'll have to bake it again.
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	244 cur_sub_entry['flags'] \|= \
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	245 SubPageFlags.FLAG_FORCED_BY_PREVIOUS_ERRORS
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	246 return STATUS_BAKE
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	247
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	248 if not prev_sub_entry:
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	249 # No previous record, so most probably was never baked. Bake it.
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	250 cur_sub_entry['flags'] \|= \
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	251 SubPageFlags.FLAG_FORCED_BY_NO_PREVIOUS
871 504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	252 return STATUS_BAKE
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	253
504ddb370df8 refactor: Fixing some issues with baking assets. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	254 return STATUS_CLEAN
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	255
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	256
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	257 def _ensure_dir_exists(path):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	258 try:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	259 os.makedirs(path, mode=0o755, exist_ok=True)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	260 except OSError:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	261 # In a multiprocess environment, several process may very
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	262 # occasionally try to create the same directory at the same time.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	263 # Let's ignore any error and if something's really wrong (like file
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	264 # acces permissions or whatever), then it will more legitimately fail
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	265 # just after this when we try to write files.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	266 pass
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	267

Mercurial > piecrust2

annotate piecrust/pipelines/_pagebaker.py @ 1051:971b4d67e82a