piecrust2: piecrust/serving/procloop.py annotate

annotate piecrust/serving/procloop.py @ 853:f070a4fc033c

core: Continue PieCrust3 refactor, simplify pages. The asset pipeline is still the only function pipeline at this point. * No more `QualifiedPage`, and several other pieces of code deleted. * Data providers are simpler and more focused. For instance, the page iterator doesn't try to support other types of items. * Route parameters are proper known source metadata to remove the confusion between the two. * Make the baker and pipeline more correctly manage records and record histories. * Add support for record collapsing and deleting stale outputs in the asset pipeline.

author	Ludovic Chabant <ludovic@chabant.com>
date	Sun, 21 May 2017 00:06:59 -0700
parents	c2ea75e37540
children	08e02c2a2a1a

rev	line source
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import os
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os.path
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import time
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import json
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import queue
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 import logging
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	7 import itertools
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8 import threading
663 3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`. Ludovic Chabant <ludovic@chabant.com> parents: 570 diff changeset	9 from piecrust import CONFIG_PATH, THEME_CONFIG_PATH
553 cc6f3dbe3048 serve: Extract some of the server's functionality into WSGI middlewares. Ludovic Chabant <ludovic@chabant.com> parents: 552 diff changeset	10 from piecrust.app import PieCrust
cc6f3dbe3048 serve: Extract some of the server's functionality into WSGI middlewares. Ludovic Chabant <ludovic@chabant.com> parents: 552 diff changeset	11 from piecrust.processing.pipeline import ProcessorPipeline
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	14 logger = logging.getLogger(__name__)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	15
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	16 # This flag is for cancelling all long running requests like SSEs.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	17 server_shutdown = False
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	20 class PipelineStatusServerSentEventProducer(object):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	21 """ The producer for Server-Sent Events (SSE) notifying the front-end
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	22 about useful things like assets having been re-processed in the
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	23 background.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	24 Each has its own queue because the user could have multiple pages
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	25 open, each having to display notifications coming from the server.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	26 """
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	27 def __init__(self, proc_loop):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	28 self._proc_loop = proc_loop
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	29 self._queue = queue.Queue()
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	30 self._start_time = 0
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	31 self._poll_interval = 0.5
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	32 self._ping_interval = 30
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	33 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	34 self._running = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	35
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	36 def addBuildEvent(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	37 self._queue.put_nowait(item)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	38
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	39 def run(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	40 logger.debug("Starting pipeline status SSE.")
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	41 self._proc_loop.addObserver(self)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	42 self._start_time = time.time()
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	43 self._running = 1
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	44
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	45 outstr = 'event: ping\ndata: started\n\n'
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	46 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	47
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	48 while self._running == 1 and not server_shutdown:
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 try:
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	50 # We use a short poll interval (less than a second) because
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	51 # we need to catch `server_shutdown` going `True` as soon as
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	52 # possible to exit this thread when the user hits `CTRL+C`.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	53 data = self._queue.get(True, self._poll_interval)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54 except queue.Empty:
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	55 # Not exact timing but close enough.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	56 self._time_between_pings += self._poll_interval
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	57 if self._time_between_pings >= self._ping_interval:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	58 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	59 logger.debug("Sending ping/heartbeat event.")
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	60 outstr = 'event: ping\ndata: 1\n\n'
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	61 yield bytes(outstr, 'utf8')
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	62 continue
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64 logger.debug("Sending pipeline status SSE.")
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	65 outstr = (('event: %s\n' % data['type']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	66 ('id: %s\n' % data['id']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	67 ('data: %s\n\n' % json.dumps(data)))
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	68 self._queue.task_done()
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	70
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71 def close(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	72 logger.debug("Closing pipeline status SSE.")
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	73 self._proc_loop.removeObserver(self)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	74 self._running = 2
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	77 class ProcessingLoop(threading.Thread):
680 c2ea75e37540 serve: Fix some crashes introduced by recent refactor. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	78 def __init__(self, appfactory, out_dir):
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	79 super(ProcessingLoop, self).__init__(
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80 name='pipeline-reloader', daemon=True)
666 81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	81 self.appfactory = appfactory
680 c2ea75e37540 serve: Fix some crashes introduced by recent refactor. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	82 self.out_dir = out_dir
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	83 self.last_status_id = 0
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84 self.interval = 1
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	85 self.app = None
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	86 self._roots = []
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	87 self._monitor_assets_root = False
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	88 self._paths = set()
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	89 self._record = None
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	90 self._last_bake = 0
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	91 self._last_config_mtime = 0
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	92 self._obs = []
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	93 self._obs_lock = threading.Lock()
680 c2ea75e37540 serve: Fix some crashes introduced by recent refactor. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	94 config_name = (
c2ea75e37540 serve: Fix some crashes introduced by recent refactor. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	95 THEME_CONFIG_PATH if appfactory.theme_site else CONFIG_PATH)
c2ea75e37540 serve: Fix some crashes introduced by recent refactor. Ludovic Chabant <ludovic@chabant.com> parents: 666 diff changeset	96 self._config_path = os.path.join(appfactory.root_dir, config_name)
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	97
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	98 def addObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	99 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	100 self._obs.append(obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	101
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	102 def removeObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	103 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	104 self._obs.remove(obs)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	105
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	106 def run(self):
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	107 self._initPipeline()
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	108
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	109 self._last_bake = time.time()
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	110 self._last_config_mtime = os.path.getmtime(self._config_path)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	111 self._record = self.pipeline.run()
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	112
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	113 while True:
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	114 cur_config_time = os.path.getmtime(self._config_path)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	115 if self._last_config_mtime < cur_config_time:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	116 logger.info("Site configuration changed, reloading pipeline.")
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	117 self._last_config_mtime = cur_config_time
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	118 self._initPipeline()
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	119 for root in self._roots:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	120 self._runPipeline(root)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	121 continue
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	122
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	123 if self._monitor_assets_root:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	124 assets_dir = os.path.join(self.app.root_dir, 'assets')
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	125 if os.path.isdir(assets_dir):
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	126 logger.info("Assets directory was created, reloading "
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	127 "pipeline.")
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	128 self._initPipeline()
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	129 self._runPipeline(assets_dir)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	130 continue
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	131
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	132 for root in self._roots:
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	133 # For each mount root we try to find the first new or
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	134 # modified file. If any, we just run the pipeline on
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	135 # that mount.
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	136 found_new_or_modified = False
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	137 for dirpath, dirnames, filenames in os.walk(root):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	138 for filename in filenames:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	139 path = os.path.join(dirpath, filename)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	140 if path not in self._paths:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	141 logger.debug("Found new asset: %s" % path)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	142 self._paths.add(path)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	143 found_new_or_modified = True
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	144 break
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	145 if os.path.getmtime(path) > self._last_bake:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	146 logger.debug("Found modified asset: %s" % path)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	147 found_new_or_modified = True
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	148 break
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	149
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150 if found_new_or_modified:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	151 break
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	152
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	153 if found_new_or_modified:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	154 self._runPipeline(root)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	155
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	156 time.sleep(self.interval)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	157
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	158 def _initPipeline(self):
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	159 # Create the app and pipeline.
666 81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business. Ludovic Chabant <ludovic@chabant.com> parents: 663 diff changeset	160 self.app = self.appfactory.create()
570 7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	161 self.pipeline = ProcessorPipeline(self.app, self.out_dir)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	162
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	163 # Get the list of assets directories.
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	164 self._roots = list(self.pipeline.mounts.keys())
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	165
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	166 # The 'assets' folder may not be in the mounts list if it doesn't
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	167 # exist yet, but we want to monitor for when the user creates it.
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	168 default_root = os.path.join(self.app.root_dir, 'assets')
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	169 self._monitor_assets_root = (default_root not in self._roots)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	170
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	171 # Build the list of initial asset files.
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	172 self._paths = set()
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	173 for root in self._roots:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	174 for dirpath, dirnames, filenames in os.walk(root):
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	175 self._paths \|= set([os.path.join(dirpath, f)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	176 for f in filenames])
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly. Ludovic Chabant <ludovic@chabant.com> parents: 565 diff changeset	177
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	178 def _runPipeline(self, root):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	179 self._last_bake = time.time()
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	180 try:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	181 self._record = self.pipeline.run(
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	182 root,
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	183 previous_record=self._record,
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	184 save_record=False)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	185
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	186 status_id = self.last_status_id + 1
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	187 self.last_status_id += 1
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	188
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	189 if self._record.success:
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	190 changed = filter(
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	191 lambda i: not i.was_collapsed_from_last_run,
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	192 self._record.entries)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	193 changed = itertools.chain.from_iterable(
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	194 map(lambda i: i.rel_outputs, changed))
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	195 changed = list(changed)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	196 item = {
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	197 'id': status_id,
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	198 'type': 'pipeline_success',
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	199 'assets': changed}
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	200
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	201 self._notifyObservers(item)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	202 else:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	203 item = {
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	204 'id': status_id,
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	205 'type': 'pipeline_error',
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	206 'assets': []}
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	207 for entry in self._record.entries:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	208 if entry.errors:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	209 asset_item = {
565 ff714d7f074d serve: Fix error reporting when the background pipeline fails. Ludovic Chabant <ludovic@chabant.com> parents: 553 diff changeset	210 'path': entry.path,
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	211 'errors': list(entry.errors)}
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	212 item['assets'].append(asset_item)
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	213
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	214 self._notifyObservers(item)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	215 except Exception as ex:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	216 logger.exception(ex)
374 fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	217
552 9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	218 def _notifyObservers(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	219 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	220 observers = list(self._obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	221 for obs in observers:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	222 obs.addBuildEvent(item)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications. Ludovic Chabant <ludovic@chabant.com> parents: 374 diff changeset	223

Mercurial > piecrust2

annotate piecrust/serving/procloop.py @ 853:f070a4fc033c