Mercurial > piecrust2

annotate piecrust/serving/procloop.py @ 1037:89d94955b818

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
serve: Fix infinite loop in asset processing when a change is detected.
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 16 Jan 2018 08:41:25 -0800
parents 7f1da7e7b154
children 54eb8ad9e809
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import time
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import json
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import queue
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 import logging
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
7 import itertools
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 import threading
663
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
9 from piecrust import CONFIG_PATH, THEME_CONFIG_PATH
880
342e3ea24b5d serve: Fix asset processing loop.
Ludovic Chabant <ludovic@chabant.com>
parents: 876
diff changeset
10 from piecrust.chefutil import format_timed_scope
1037
89d94955b818 serve: Fix infinite loop in asset processing when a change is detected.
Ludovic Chabant <ludovic@chabant.com>
parents: 918
diff changeset
11 from piecrust.pipelines.records import MultiRecord
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 logger = logging.getLogger(__name__)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
16 # This flag is for cancelling all long running requests like SSEs.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
17 server_shutdown = False
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
20 class PipelineStatusServerSentEventProducer(object):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
21 """ The producer for Server-Sent Events (SSE) notifying the front-end
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
22 about useful things like assets having been re-processed in the
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
23 background.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
24 Each has its own queue because the user could have multiple pages
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
25 open, each having to display notifications coming from the server.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
26 """
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
27 def __init__(self, proc_loop):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
28 self._proc_loop = proc_loop
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
29 self._queue = queue.Queue()
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 self._start_time = 0
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
31 self._poll_interval = 0.5
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
32 self._ping_interval = 30
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
33 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
34 self._running = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
35
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
36 def addBuildEvent(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
37 self._queue.put_nowait(item)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 def run(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 logger.debug("Starting pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
41 self._proc_loop.addObserver(self)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 self._start_time = time.time()
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
43 self._running = 1
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 outstr = 'event: ping\ndata: started\n\n'
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
48 while self._running == 1 and not server_shutdown:
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 try:
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
50 # We use a short poll interval (less than a second) because
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
51 # we need to catch `server_shutdown` going `True` as soon as
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
52 # possible to exit this thread when the user hits `CTRL+C`.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
53 data = self._queue.get(True, self._poll_interval)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 except queue.Empty:
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
55 # Not exact timing but close enough.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
56 self._time_between_pings += self._poll_interval
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
57 if self._time_between_pings >= self._ping_interval:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
58 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
59 logger.debug("Sending ping/heartbeat event.")
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
60 outstr = 'event: ping\ndata: 1\n\n'
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
61 yield bytes(outstr, 'utf8')
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
62 continue
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 logger.debug("Sending pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
65 outstr = (('event: %s\n' % data['type']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
66 ('id: %s\n' % data['id']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
67 ('data: %s\n\n' % json.dumps(data)))
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
68 self._queue.task_done()
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 def close(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 logger.debug("Closing pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
73 self._proc_loop.removeObserver(self)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
74 self._running = 2
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
77 class _AssetProcessingInfo:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
78 def __init__(self, source):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
79 self.source = source
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
80 self.paths = set()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
81 self.last_bake_time = time.time()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
82
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
83
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84 class ProcessingLoop(threading.Thread):
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
85 def __init__(self, appfactory, out_dir):
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
86 super().__init__(name='pipeline-reloader', daemon=True)
666
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business.
Ludovic Chabant <ludovic@chabant.com>
parents: 663
diff changeset
87 self.appfactory = appfactory
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
88 self.out_dir = out_dir
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
89 self.last_status_id = 0
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
90 self.interval = 1
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
91 self._app = None
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
92 self._proc_infos = None
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
93 self._last_records = None
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
94 self._last_config_mtime = 0
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
95 self._obs = []
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
96 self._obs_lock = threading.Lock()
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
97 config_name = (
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
98 THEME_CONFIG_PATH if appfactory.theme_site else CONFIG_PATH)
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
99 self._config_path = os.path.join(appfactory.root_dir, config_name)
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
100
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
101 def addObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
102 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
103 self._obs.append(obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
104
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
105 def removeObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
106 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
107 self._obs.remove(obs)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
108
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
109 def run(self):
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
110 logger.debug("Initializing processing loop with output: %s" %
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
111 self.out_dir)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
112 try:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
113 self._init()
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
114 except Exception as ex:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
115 logger.error("Error initializing processing loop:")
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
116 logger.exception(ex)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
117 return
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
118
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
119 logger.debug("Doing initial processing loop bake...")
918
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
120 self._runPipelinesSafe()
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
121
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
122 logger.debug("Running processing loop...")
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
123 self._last_config_mtime = os.path.getmtime(self._config_path)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
124
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
125 while True:
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
126 cur_config_time = os.path.getmtime(self._config_path)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
127 if self._last_config_mtime < cur_config_time:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
128 logger.info("Site configuration changed, reloading pipeline.")
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
129 self._last_config_mtime = cur_config_time
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
130 self._init()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
131 self._runPipelines()
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
132 continue
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
133
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
134 for procinfo in self._proc_infos.values():
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
135 # For each assets folder we try to find the first new or
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
136 # modified file. If any, we just run the pipeline on
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
137 # that source.
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
138 found_new_or_modified = False
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
139 for item in procinfo.source.getAllContents():
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
140 path = item.spec
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
141 if path not in procinfo.paths:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
142 logger.debug("Found new asset: %s" % path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
143 procinfo.paths.add(path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
144 found_new_or_modified = True
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
145 break
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
146 if os.path.getmtime(path) > procinfo.last_bake_time:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
147 logger.debug("Found modified asset: %s" % path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
148 found_new_or_modified = True
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
149 break
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
150 if found_new_or_modified:
1037
89d94955b818 serve: Fix infinite loop in asset processing when a change is detected.
Ludovic Chabant <ludovic@chabant.com>
parents: 918
diff changeset
151 logger.info("change detected, reprocessed '%s'." %
89d94955b818 serve: Fix infinite loop in asset processing when a change is detected.
Ludovic Chabant <ludovic@chabant.com>
parents: 918
diff changeset
152 procinfo.source.name)
89d94955b818 serve: Fix infinite loop in asset processing when a change is detected.
Ludovic Chabant <ludovic@chabant.com>
parents: 918
diff changeset
153 self._runPipelinesSafe(procinfo.source)
89d94955b818 serve: Fix infinite loop in asset processing when a change is detected.
Ludovic Chabant <ludovic@chabant.com>
parents: 918
diff changeset
154 procinfo.last_bake_time = time.time()
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
155
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
156 time.sleep(self.interval)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
157
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
158 def _init(self):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
159 self._app = self.appfactory.create()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
160 self._last_records = MultiRecord()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
161
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
162 self._proc_infos = {}
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
163 for src in self._app.sources:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
164 if src.config['pipeline'] != 'asset':
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
165 continue
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
166
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
167 procinfo = _AssetProcessingInfo(src)
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
168 self._proc_infos[src.name] = procinfo
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
169
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
170 # Build the list of initial asset files.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
171 for item in src.getAllContents():
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
172 procinfo.paths.add(item.spec)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
173
918
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
174 def _runPipelinesSafe(self, only_for_source=None):
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
175 try:
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
176 self._runPipelines(only_for_source)
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
177 except Exception as ex:
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
178 logger.error("Error while running asset pipeline:")
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
179 logger.exception(ex)
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
180
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
181 def _runPipelines(self, only_for_source):
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
182 from piecrust.baking.baker import Baker
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
183
918
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
184 allowed_sources = None
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
185 if only_for_source:
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
186 allowed_sources = [only_for_source.name]
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
187 baker = Baker(
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
188 self.appfactory, self._app, self.out_dir,
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
189 allowed_pipelines=['asset'],
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
190 allowed_sources=allowed_sources,
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
191 rotate_bake_records=False)
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
192 records = baker.bake()
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
193
918
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
194 self._onPipelinesRun(records)
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
195
918
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
196 def _onPipelinesRun(self, records):
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
197 self.last_status_id += 1
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
198
918
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
199 if records.success:
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
200 for rec in records.records:
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
201 changed = filter(
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
202 lambda i: not i.was_collapsed_from_last_run,
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
203 rec.getEntries())
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
204 changed = itertools.chain.from_iterable(
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
205 map(lambda i: i.out_paths, changed))
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
206 changed = list(changed)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
207 item = {
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
208 'id': self.last_status_id,
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
209 'type': 'pipeline_success',
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
210 'assets': changed}
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
211
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
212 self._notifyObservers(item)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
213 else:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
214 item = {
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
215 'id': self.last_status_id,
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
216 'type': 'pipeline_error',
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
217 'assets': []}
918
7f1da7e7b154 internal: The processing loop for the server is now using the baker.
Ludovic Chabant <ludovic@chabant.com>
parents: 880
diff changeset
218 for rec in records.records:
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
219 for entry in rec.getEntries():
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
220 if entry.errors:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
221 asset_item = {
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
222 'path': entry.item_spec,
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
223 'errors': list(entry.errors)}
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
224 item['assets'].append(asset_item)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
225
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
226 self._notifyObservers(item)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
227
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
228 def _notifyObservers(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
229 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
230 observers = list(self._obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
231 for obs in observers:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
232 obs.addBuildEvent(item)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
233