Mercurial > piecrust2

annotate piecrust/serving/procloop.py @ 663:3ceeca7bb71c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
themes: Add support for a `--theme` argument to `chef`.
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 01 Mar 2016 22:27:28 -0800
parents 7dabfdd056a1
children 81d9c3a3a0b5
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import time
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import json
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import queue
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 import logging
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
7 import itertools
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 import threading
663
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
9 from piecrust import CONFIG_PATH, THEME_CONFIG_PATH
553
cc6f3dbe3048 serve: Extract some of the server's functionality into WSGI middlewares.
Ludovic Chabant <ludovic@chabant.com>
parents: 552
diff changeset
10 from piecrust.app import PieCrust
cc6f3dbe3048 serve: Extract some of the server's functionality into WSGI middlewares.
Ludovic Chabant <ludovic@chabant.com>
parents: 552
diff changeset
11 from piecrust.processing.pipeline import ProcessorPipeline
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 logger = logging.getLogger(__name__)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
16 # This flag is for cancelling all long running requests like SSEs.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
17 server_shutdown = False
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
20 class PipelineStatusServerSentEventProducer(object):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
21 """ The producer for Server-Sent Events (SSE) notifying the front-end
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
22 about useful things like assets having been re-processed in the
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
23 background.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
24 Each has its own queue because the user could have multiple pages
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
25 open, each having to display notifications coming from the server.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
26 """
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
27 def __init__(self, proc_loop):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
28 self._proc_loop = proc_loop
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
29 self._queue = queue.Queue()
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 self._start_time = 0
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
31 self._poll_interval = 0.5
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
32 self._ping_interval = 30
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
33 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
34 self._running = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
35
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
36 def addBuildEvent(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
37 self._queue.put_nowait(item)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 def run(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 logger.debug("Starting pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
41 self._proc_loop.addObserver(self)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 self._start_time = time.time()
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
43 self._running = 1
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 outstr = 'event: ping\ndata: started\n\n'
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
48 while self._running == 1 and not server_shutdown:
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 try:
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
50 # We use a short poll interval (less than a second) because
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
51 # we need to catch `server_shutdown` going `True` as soon as
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
52 # possible to exit this thread when the user hits `CTRL+C`.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
53 data = self._queue.get(True, self._poll_interval)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 except queue.Empty:
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
55 # Not exact timing but close enough.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
56 self._time_between_pings += self._poll_interval
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
57 if self._time_between_pings >= self._ping_interval:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
58 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
59 logger.debug("Sending ping/heartbeat event.")
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
60 outstr = 'event: ping\ndata: 1\n\n'
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
61 yield bytes(outstr, 'utf8')
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
62 continue
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 logger.debug("Sending pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
65 outstr = (('event: %s\n' % data['type']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
66 ('id: %s\n' % data['id']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
67 ('data: %s\n\n' % json.dumps(data)))
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
68 self._queue.task_done()
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 def close(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 logger.debug("Closing pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
73 self._proc_loop.removeObserver(self)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
74 self._running = 2
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 class ProcessingLoop(threading.Thread):
663
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
78 def __init__(self, root_dir, out_dir, sub_cache_dir=None,
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
79 theme_site=False, debug=False):
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 super(ProcessingLoop, self).__init__(
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81 name='pipeline-reloader', daemon=True)
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
82 self.root_dir = root_dir
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
83 self.out_dir = out_dir
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
84 self.sub_cache_dir = sub_cache_dir
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
85 self.debug = debug
663
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
86 self.theme_site = theme_site
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
87 self.last_status_id = 0
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
88 self.interval = 1
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
89 self.app = None
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
90 self._roots = []
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
91 self._monitor_assets_root = False
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
92 self._paths = set()
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
93 self._record = None
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
94 self._last_bake = 0
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
95 self._last_config_mtime = 0
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
96 self._obs = []
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
97 self._obs_lock = threading.Lock()
663
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
98 if theme_site:
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
99 self._config_path = os.path.join(root_dir, THEME_CONFIG_PATH)
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
100 else:
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
101 self._config_path = os.path.join(root_dir, CONFIG_PATH)
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
102
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
103 def addObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
104 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
105 self._obs.append(obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
106
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
107 def removeObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
108 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
109 self._obs.remove(obs)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
110
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
111 def run(self):
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
112 self._initPipeline()
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
113
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
114 self._last_bake = time.time()
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
115 self._last_config_mtime = os.path.getmtime(self._config_path)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
116 self._record = self.pipeline.run()
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
117
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
118 while True:
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
119 cur_config_time = os.path.getmtime(self._config_path)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
120 if self._last_config_mtime < cur_config_time:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
121 logger.info("Site configuration changed, reloading pipeline.")
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
122 self._last_config_mtime = cur_config_time
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
123 self._initPipeline()
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
124 for root in self._roots:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
125 self._runPipeline(root)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
126 continue
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
127
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
128 if self._monitor_assets_root:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
129 assets_dir = os.path.join(self.app.root_dir, 'assets')
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
130 if os.path.isdir(assets_dir):
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
131 logger.info("Assets directory was created, reloading "
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
132 "pipeline.")
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
133 self._initPipeline()
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
134 self._runPipeline(assets_dir)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
135 continue
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
136
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
137 for root in self._roots:
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
138 # For each mount root we try to find the first new or
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
139 # modified file. If any, we just run the pipeline on
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
140 # that mount.
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
141 found_new_or_modified = False
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
142 for dirpath, dirnames, filenames in os.walk(root):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
143 for filename in filenames:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
144 path = os.path.join(dirpath, filename)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
145 if path not in self._paths:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
146 logger.debug("Found new asset: %s" % path)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
147 self._paths.add(path)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
148 found_new_or_modified = True
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
149 break
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
150 if os.path.getmtime(path) > self._last_bake:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
151 logger.debug("Found modified asset: %s" % path)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
152 found_new_or_modified = True
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
153 break
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
154
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
155 if found_new_or_modified:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
156 break
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
157
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
158 if found_new_or_modified:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
159 self._runPipeline(root)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
160
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
161 time.sleep(self.interval)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
162
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
163 def _initPipeline(self):
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
164 # Create the app and pipeline.
663
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
165 self.app = PieCrust(root_dir=self.root_dir, debug=self.debug,
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
166 theme_site=self.theme_site)
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
167 if self.sub_cache_dir:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
168 self.app._useSubCacheDir(self.sub_cache_dir)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
169 self.pipeline = ProcessorPipeline(self.app, self.out_dir)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
170
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
171 # Get the list of assets directories.
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
172 self._roots = list(self.pipeline.mounts.keys())
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
173
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
174 # The 'assets' folder may not be in the mounts list if it doesn't
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
175 # exist yet, but we want to monitor for when the user creates it.
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
176 default_root = os.path.join(self.app.root_dir, 'assets')
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
177 self._monitor_assets_root = (default_root not in self._roots)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
178
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
179 # Build the list of initial asset files.
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
180 self._paths = set()
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
181 for root in self._roots:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
182 for dirpath, dirnames, filenames in os.walk(root):
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
183 self._paths |= set([os.path.join(dirpath, f)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
184 for f in filenames])
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
185
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
186 def _runPipeline(self, root):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
187 self._last_bake = time.time()
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
188 try:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
189 self._record = self.pipeline.run(
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
190 root,
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
191 previous_record=self._record,
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
192 save_record=False)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
193
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
194 status_id = self.last_status_id + 1
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
195 self.last_status_id += 1
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
196
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
197 if self._record.success:
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
198 changed = filter(
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
199 lambda i: not i.was_collapsed_from_last_run,
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
200 self._record.entries)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
201 changed = itertools.chain.from_iterable(
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
202 map(lambda i: i.rel_outputs, changed))
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
203 changed = list(changed)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
204 item = {
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
205 'id': status_id,
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
206 'type': 'pipeline_success',
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
207 'assets': changed}
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
208
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
209 self._notifyObservers(item)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
210 else:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
211 item = {
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
212 'id': status_id,
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
213 'type': 'pipeline_error',
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
214 'assets': []}
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
215 for entry in self._record.entries:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
216 if entry.errors:
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
217 asset_item = {
565
ff714d7f074d serve: Fix error reporting when the background pipeline fails.
Ludovic Chabant <ludovic@chabant.com>
parents: 553
diff changeset
218 'path': entry.path,
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
219 'errors': list(entry.errors)}
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
220 item['assets'].append(asset_item)
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
221
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
222 self._notifyObservers(item)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
223 except Exception as ex:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
224 logger.exception(ex)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
225
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
226 def _notifyObservers(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
227 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
228 observers = list(self._obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
229 for obs in observers:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
230 obs.addBuildEvent(item)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
231