piecrust2: piecrust/serving/procloop.py annotate

annotate piecrust/serving/procloop.py @ 918:7f1da7e7b154

internal: The processing loop for the server is now using the baker. Instead of reimplementing a custom way to run the pipelines, the loop is just calling the baker, but only for asset pipelines.

author	Ludovic Chabant <ludovic@chabant.com>
date	Fri, 29 Sep 2017 08:43:34 -0700
parents	342e3ea24b5d
children	89d94955b818

rev	line source
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import os
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os.path
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import time
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import json
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import queue
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 import logging
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	7 import itertools
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8 import threading
663 3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`. Ludovic Chabant <ludovic@chabant.com> parents: 570 diff changeset	9 from piecrust import CONFIG_PATH, THEME_CONFIG_PATH
880 342e3ea24b5d serve: Fix asset processing loop. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	10 from piecrust.chefutil import format_timed_scope
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	11 from piecrust.pipelines.base import (
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	12 PipelineJobCreateContext, PipelineJobRunContext, PipelineJobResult,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	13 PipelineManager)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	14 from piecrust.pipelines.records import (
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	15 MultiRecord, MultiRecordHistory)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18 logger = logging.getLogger(__name__)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	20 # This flag is for cancelling all long running requests like SSEs.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	21 server_shutdown = False
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	23
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	24 class PipelineStatusServerSentEventProducer(object):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	25 """ The producer for Server-Sent Events (SSE) notifying the front-end
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	26 about useful things like assets having been re-processed in the
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	27 background.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	28 Each has its own queue because the user could have multiple pages
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	29 open, each having to display notifications coming from the server.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	30 """
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	31 def __init__(self, proc_loop):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	32 self._proc_loop = proc_loop
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	33 self._queue = queue.Queue()
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34 self._start_time = 0
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	35 self._poll_interval = 0.5
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	36 self._ping_interval = 30
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	37 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	38 self._running = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	39
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	40 def addBuildEvent(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	41 self._queue.put_nowait(item)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	43 def run(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44 logger.debug("Starting pipeline status SSE.")
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	45 self._proc_loop.addObserver(self)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	46 self._start_time = time.time()
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	47 self._running = 1
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	48
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 outstr = 'event: ping\ndata: started\n\n'
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	50 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	52 while self._running == 1 and not server_shutdown:
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53 try:
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	54 # We use a short poll interval (less than a second) because
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	55 # we need to catch `server_shutdown` going `True` as soon as
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	56 # possible to exit this thread when the user hits `CTRL+C`.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	57 data = self._queue.get(True, self._poll_interval)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58 except queue.Empty:
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	59 # Not exact timing but close enough.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	60 self._time_between_pings += self._poll_interval
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	61 if self._time_between_pings >= self._ping_interval:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	62 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	63 logger.debug("Sending ping/heartbeat event.")
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	64 outstr = 'event: ping\ndata: 1\n\n'
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	65 yield bytes(outstr, 'utf8')
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	66 continue
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	67
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 logger.debug("Sending pipeline status SSE.")
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	69 outstr = (('event: %s\n' % data['type']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	70 ('id: %s\n' % data['id']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	71 ('data: %s\n\n' % json.dumps(data)))
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	72 self._queue.task_done()
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	73 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	74
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75 def close(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76 logger.debug("Closing pipeline status SSE.")
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	77 self._proc_loop.removeObserver(self)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	78 self._running = 2
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	79
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	81 class _AssetProcessingInfo:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	82 def __init__(self, source):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	83 self.source = source
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	84 self.paths = set()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	85 self.last_bake_time = time.time()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	86
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	87
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	88 class ProcessingLoop(threading.Thread):
680 c2ea75e37540 serve: Fix some crashes introduced by recent refactor. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	89 def __init__(self, appfactory, out_dir):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	90 super().__init__(name='pipeline-reloader', daemon=True)
666 81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	91 self.appfactory = appfactory
680 c2ea75e37540 serve: Fix some crashes introduced by recent refactor. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	92 self.out_dir = out_dir
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	93 self.last_status_id = 0
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	94 self.interval = 1
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	95 self._app = None
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	96 self._proc_infos = None
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	97 self._last_records = None
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	98 self._last_config_mtime = 0
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	99 self._obs = []
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	100 self._obs_lock = threading.Lock()
680 c2ea75e37540 serve: Fix some crashes introduced by recent refactor. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	101 config_name = (
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	102 THEME_CONFIG_PATH if appfactory.theme_site else CONFIG_PATH)
680 c2ea75e37540 serve: Fix some crashes introduced by recent refactor. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	103 self._config_path = os.path.join(appfactory.root_dir, config_name)
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	104
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	105 def addObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	106 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	107 self._obs.append(obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	108
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	109 def removeObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	110 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	111 self._obs.remove(obs)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	112
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	113 def run(self):
860 c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	114 logger.debug("Initializing processing loop with output: %s" %
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	115 self.out_dir)
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	116 try:
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	117 self._init()
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	118 except Exception as ex:
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	119 logger.error("Error initializing processing loop:")
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	120 logger.exception(ex)
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	121 return
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	122
860 c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	123 logger.debug("Doing initial processing loop bake...")
918 7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	124 self._runPipelinesSafe()
860 c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	125
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	126 logger.debug("Running processing loop...")
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	127 self._last_config_mtime = os.path.getmtime(self._config_path)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	128
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	129 while True:
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	130 cur_config_time = os.path.getmtime(self._config_path)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	131 if self._last_config_mtime < cur_config_time:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	132 logger.info("Site configuration changed, reloading pipeline.")
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	133 self._last_config_mtime = cur_config_time
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	134 self._init()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	135 self._runPipelines()
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	136 continue
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	137
860 c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	138 for procinfo in self._proc_infos.values():
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	139 # For each assets folder we try to find the first new or
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	140 # modified file. If any, we just run the pipeline on
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	141 # that source.
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	142 found_new_or_modified = False
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	143 for item in procinfo.source.getAllContents():
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	144 path = item.spec
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	145 if path not in procinfo.paths:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	146 logger.debug("Found new asset: %s" % path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	147 procinfo.paths.add(path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	148 found_new_or_modified = True
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	149 break
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	150 if os.path.getmtime(path) > procinfo.last_bake_time:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	151 logger.debug("Found modified asset: %s" % path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	152 found_new_or_modified = True
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	153 break
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	154 if found_new_or_modified:
880 342e3ea24b5d serve: Fix asset processing loop. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	155 with format_timed_scope(
342e3ea24b5d serve: Fix asset processing loop. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	156 logger,
342e3ea24b5d serve: Fix asset processing loop. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	157 "change detected, reprocessed '%s'." %
342e3ea24b5d serve: Fix asset processing loop. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	158 procinfo.source.name):
918 7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	159 self._runPipelinesSafe(procinfo.source)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	160
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	161 time.sleep(self.interval)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	162
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	163 def _init(self):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	164 self._app = self.appfactory.create()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	165 self._last_records = MultiRecord()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	166
860 c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	167 self._proc_infos = {}
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	168 for src in self._app.sources:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	169 if src.config['pipeline'] != 'asset':
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	170 continue
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	171
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	172 procinfo = _AssetProcessingInfo(src)
860 c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	173 self._proc_infos[src.name] = procinfo
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	174
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	175 # Build the list of initial asset files.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	176 for item in src.getAllContents():
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	177 procinfo.paths.add(item.spec)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	178
918 7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	179 def _runPipelinesSafe(self, only_for_source=None):
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	180 try:
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	181 self._runPipelines(only_for_source)
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	182 except Exception as ex:
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	183 logger.error("Error while running asset pipeline:")
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	184 logger.exception(ex)
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	185
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	186 def _runPipelines(self, only_for_source):
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	187 from piecrust.baking.baker import Baker
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	188
918 7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	189 allowed_sources = None
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	190 if only_for_source:
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	191 allowed_sources = [only_for_source.name]
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	192 baker = Baker(
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	193 self.appfactory, self._app, self.out_dir,
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	194 allowed_pipelines=['asset'],
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	195 allowed_sources=allowed_sources,
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	196 rotate_bake_records=False)
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	197 records = baker.bake()
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	198
918 7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	199 self._onPipelinesRun(records)
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	200
918 7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	201 def _onPipelinesRun(self, records):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	202 self.last_status_id += 1
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	203
918 7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	204 if records.success:
7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	205 for rec in records.records:
860 c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	206 changed = filter(
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	207 lambda i: not i.was_collapsed_from_last_run,
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	208 rec.getEntries())
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	209 changed = itertools.chain.from_iterable(
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	210 map(lambda i: i.out_paths, changed))
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	211 changed = list(changed)
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	212 item = {
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	213 'id': self.last_status_id,
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	214 'type': 'pipeline_success',
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	215 'assets': changed}
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	216
860 c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	217 self._notifyObservers(item)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	218 else:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	219 item = {
860 c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	220 'id': self.last_status_id,
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	221 'type': 'pipeline_error',
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	222 'assets': []}
918 7f1da7e7b154 internal: The processing loop for the server is now using the baker. Ludovic Chabant <ludovic@chabant.com> parents: 880 diff changeset	223 for rec in records.records:
860 c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	224 for entry in rec.getEntries():
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	225 if entry.errors:
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	226 asset_item = {
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	227 'path': entry.item_spec,
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	228 'errors': list(entry.errors)}
c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	229 item['assets'].append(asset_item)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	230
860 c71472e6537f refactor: Get the processing loop in the server functional again. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	231 self._notifyObservers(item)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 680 diff changeset	232
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	233 def _notifyObservers(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	234 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	235 observers = list(self._obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	236 for obs in observers:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	237 obs.addBuildEvent(item)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	238

Mercurial > piecrust2

annotate piecrust/serving/procloop.py @ 918:7f1da7e7b154