Mercurial > piecrust2

annotate piecrust/serving/procloop.py @ 895:accfe8fc8440

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
admin: Add a Micropub endpoint.
author Ludovic Chabant <ludovic@chabant.com>
date Sun, 02 Jul 2017 22:23:12 -0700
parents 342e3ea24b5d
children 7f1da7e7b154
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import time
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import json
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import queue
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 import logging
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
7 import itertools
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 import threading
663
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
9 from piecrust import CONFIG_PATH, THEME_CONFIG_PATH
880
342e3ea24b5d serve: Fix asset processing loop.
Ludovic Chabant <ludovic@chabant.com>
parents: 876
diff changeset
10 from piecrust.chefutil import format_timed_scope
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
11 from piecrust.pipelines.base import (
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
12 PipelineJobCreateContext, PipelineJobRunContext, PipelineJobResult,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
13 PipelineManager)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
14 from piecrust.pipelines.records import (
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
15 MultiRecord, MultiRecordHistory)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 logger = logging.getLogger(__name__)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
20 # This flag is for cancelling all long running requests like SSEs.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
21 server_shutdown = False
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
24 class PipelineStatusServerSentEventProducer(object):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
25 """ The producer for Server-Sent Events (SSE) notifying the front-end
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
26 about useful things like assets having been re-processed in the
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
27 background.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
28 Each has its own queue because the user could have multiple pages
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
29 open, each having to display notifications coming from the server.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
30 """
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
31 def __init__(self, proc_loop):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
32 self._proc_loop = proc_loop
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
33 self._queue = queue.Queue()
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 self._start_time = 0
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
35 self._poll_interval = 0.5
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
36 self._ping_interval = 30
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
37 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
38 self._running = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
39
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
40 def addBuildEvent(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
41 self._queue.put_nowait(item)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 def run(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 logger.debug("Starting pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
45 self._proc_loop.addObserver(self)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 self._start_time = time.time()
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
47 self._running = 1
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 outstr = 'event: ping\ndata: started\n\n'
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
52 while self._running == 1 and not server_shutdown:
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 try:
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
54 # We use a short poll interval (less than a second) because
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
55 # we need to catch `server_shutdown` going `True` as soon as
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
56 # possible to exit this thread when the user hits `CTRL+C`.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
57 data = self._queue.get(True, self._poll_interval)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 except queue.Empty:
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
59 # Not exact timing but close enough.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
60 self._time_between_pings += self._poll_interval
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
61 if self._time_between_pings >= self._ping_interval:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
62 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
63 logger.debug("Sending ping/heartbeat event.")
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
64 outstr = 'event: ping\ndata: 1\n\n'
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
65 yield bytes(outstr, 'utf8')
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
66 continue
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68 logger.debug("Sending pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
69 outstr = (('event: %s\n' % data['type']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
70 ('id: %s\n' % data['id']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
71 ('data: %s\n\n' % json.dumps(data)))
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
72 self._queue.task_done()
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 def close(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 logger.debug("Closing pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
77 self._proc_loop.removeObserver(self)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
78 self._running = 2
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
81 class _AssetProcessingInfo:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
82 def __init__(self, source):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
83 self.source = source
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
84 self.paths = set()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
85 self.last_bake_time = time.time()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
86
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
87
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
88 class ProcessingLoop(threading.Thread):
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
89 def __init__(self, appfactory, out_dir):
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
90 super().__init__(name='pipeline-reloader', daemon=True)
666
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business.
Ludovic Chabant <ludovic@chabant.com>
parents: 663
diff changeset
91 self.appfactory = appfactory
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
92 self.out_dir = out_dir
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
93 self.last_status_id = 0
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
94 self.interval = 1
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
95 self._app = None
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
96 self._proc_infos = None
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
97 self._last_records = None
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
98 self._last_config_mtime = 0
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
99 self._obs = []
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
100 self._obs_lock = threading.Lock()
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
101 config_name = (
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
102 THEME_CONFIG_PATH if appfactory.theme_site else CONFIG_PATH)
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
103 self._config_path = os.path.join(appfactory.root_dir, config_name)
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
104
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
105 def addObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
106 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
107 self._obs.append(obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
108
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
109 def removeObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
110 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
111 self._obs.remove(obs)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
112
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
113 def run(self):
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
114 logger.debug("Initializing processing loop with output: %s" %
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
115 self.out_dir)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
116 try:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
117 self._init()
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
118 except Exception as ex:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
119 logger.error("Error initializing processing loop:")
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
120 logger.exception(ex)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
121 return
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
122
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
123 logger.debug("Doing initial processing loop bake...")
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
124 self._runPipelines()
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
125
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
126 logger.debug("Running processing loop...")
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
127 self._last_config_mtime = os.path.getmtime(self._config_path)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
128
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
129 while True:
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
130 cur_config_time = os.path.getmtime(self._config_path)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
131 if self._last_config_mtime < cur_config_time:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
132 logger.info("Site configuration changed, reloading pipeline.")
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
133 self._last_config_mtime = cur_config_time
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
134 self._init()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
135 self._runPipelines()
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
136 continue
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
137
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
138 for procinfo in self._proc_infos.values():
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
139 # For each assets folder we try to find the first new or
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
140 # modified file. If any, we just run the pipeline on
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
141 # that source.
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
142 found_new_or_modified = False
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
143 for item in procinfo.source.getAllContents():
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
144 path = item.spec
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
145 if path not in procinfo.paths:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
146 logger.debug("Found new asset: %s" % path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
147 procinfo.paths.add(path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
148 found_new_or_modified = True
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
149 break
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
150 if os.path.getmtime(path) > procinfo.last_bake_time:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
151 logger.debug("Found modified asset: %s" % path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
152 found_new_or_modified = True
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
153 break
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
154 if found_new_or_modified:
880
342e3ea24b5d serve: Fix asset processing loop.
Ludovic Chabant <ludovic@chabant.com>
parents: 876
diff changeset
155 with format_timed_scope(
342e3ea24b5d serve: Fix asset processing loop.
Ludovic Chabant <ludovic@chabant.com>
parents: 876
diff changeset
156 logger,
342e3ea24b5d serve: Fix asset processing loop.
Ludovic Chabant <ludovic@chabant.com>
parents: 876
diff changeset
157 "change detected, reprocessed '%s'." %
342e3ea24b5d serve: Fix asset processing loop.
Ludovic Chabant <ludovic@chabant.com>
parents: 876
diff changeset
158 procinfo.source.name):
342e3ea24b5d serve: Fix asset processing loop.
Ludovic Chabant <ludovic@chabant.com>
parents: 876
diff changeset
159 self._runPipelines(procinfo.source)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
160
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
161 time.sleep(self.interval)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
162
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
163 def _init(self):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
164 self._app = self.appfactory.create()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
165 self._last_records = MultiRecord()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
166
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
167 self._proc_infos = {}
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
168 for src in self._app.sources:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
169 if src.config['pipeline'] != 'asset':
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
170 continue
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
171
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
172 procinfo = _AssetProcessingInfo(src)
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
173 self._proc_infos[src.name] = procinfo
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
174
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
175 # Build the list of initial asset files.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
176 for item in src.getAllContents():
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
177 procinfo.paths.add(item.spec)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
178
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
179 def _runPipelines(self, only_for_source=None):
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
180 current_records = MultiRecord()
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
181 record_histories = MultiRecordHistory(
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
182 self._last_records, current_records)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
183 ppmngr = PipelineManager(
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
184 self._app, self.out_dir, record_histories)
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
185
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
186 # Create the pipelines, but also remember some stuff for what
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
187 # we want to do.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
188 for src in self._app.sources:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
189 if src.config['pipeline'] != 'asset':
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
190 continue
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
191 if only_for_source is not None and src != only_for_source:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
192 continue
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
193
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
194 ppmngr.createPipeline(src)
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
195
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
196 for ppinfo in ppmngr.getPipelines():
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
197 self._runPipeline(ppmngr, ppinfo)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
198
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
199 self.last_status_id += 1
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
200
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
201 if self._last_records.success:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
202 for rec in self._last_records.records:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
203 changed = filter(
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
204 lambda i: not i.was_collapsed_from_last_run,
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
205 rec.getEntries())
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
206 changed = itertools.chain.from_iterable(
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
207 map(lambda i: i.out_paths, changed))
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
208 changed = list(changed)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
209 item = {
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
210 'id': self.last_status_id,
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
211 'type': 'pipeline_success',
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
212 'assets': changed}
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
213
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
214 self._notifyObservers(item)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
215 else:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
216 item = {
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
217 'id': self.last_status_id,
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
218 'type': 'pipeline_error',
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
219 'assets': []}
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
220 for rec in self._last_records.records:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
221 for entry in rec.getEntries():
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
222 if entry.errors:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
223 asset_item = {
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
224 'path': entry.item_spec,
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
225 'errors': list(entry.errors)}
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
226 item['assets'].append(asset_item)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
227
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
228 self._notifyObservers(item)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
229
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
230 def _runPipeline(self, ppmngr, ppinfo):
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
231 src = ppinfo.source
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
232 logger.debug("Running pipeline '%s' on: %s" %
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
233 (ppinfo.pipeline_name, src.name))
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
234
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
235 # Set the time.
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
236 procinfo = self._proc_infos[src.name]
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
237 procinfo.last_bake_time = time.time()
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
238
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
239 # Process all items in the source.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
240 pp = ppinfo.pipeline
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
241 cr = ppinfo.record_history.current
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
242 record_histories = ppmngr.record_histories
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
243 current_records = record_histories.current
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
244 jobctx = PipelineJobCreateContext(0, record_histories)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
245 jobs = pp.createJobs(jobctx)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
246 for job in jobs:
876
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters.
Ludovic Chabant <ludovic@chabant.com>
parents: 860
diff changeset
247 runctx = PipelineJobRunContext(
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters.
Ludovic Chabant <ludovic@chabant.com>
parents: 860
diff changeset
248 job, pp.record_name, record_histories)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
249
876
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters.
Ludovic Chabant <ludovic@chabant.com>
parents: 860
diff changeset
250 ppres = PipelineJobResult()
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters.
Ludovic Chabant <ludovic@chabant.com>
parents: 860
diff changeset
251 ppres.record_entry = pp.createRecordEntry(job, runctx)
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters.
Ludovic Chabant <ludovic@chabant.com>
parents: 860
diff changeset
252
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
253 try:
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
254 pp.run(job, runctx, ppres)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
255 except Exception as e:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
256 ppres.record_entry.errors.append(str(e))
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
257
880
342e3ea24b5d serve: Fix asset processing loop.
Ludovic Chabant <ludovic@chabant.com>
parents: 876
diff changeset
258 if ppres.next_step_job is not None:
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
259 logger.error("The processing loop for the server "
880
342e3ea24b5d serve: Fix asset processing loop.
Ludovic Chabant <ludovic@chabant.com>
parents: 876
diff changeset
260 "doesn't support multi-step pipelines.")
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
261
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
262 cr.addEntry(ppres.record_entry)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
263 if not ppres.record_entry.success:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
264 cr.success = False
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
265 current_records.success = False
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
266 logger.error("Errors found in %s:" % job.content_item.spec)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
267 for e in ppres.record_entry.errors:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
268 logger.error(" " + e)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
269
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
270 # Do all the final stuff.
855
448710d84121 refactor: Get the taxonomy support back to a functional state.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
271 ppmngr.postJobRun()
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
272 ppmngr.deleteStaleOutputs()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
273 ppmngr.collapseRecords()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
274 ppmngr.shutdownPipelines()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
275
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
276 # Swap the old record with the next record.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
277 pr = ppinfo.record_history.previous
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
278 logger.debug("Swapping record '%s' with '%s'." % (pr.name, cr.name))
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
279 self._last_records.records.remove(pr)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
280 self._last_records.records.append(cr)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
281
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
282 def _notifyObservers(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
283 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
284 observers = list(self._obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
285 for obs in observers:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
286 obs.addBuildEvent(item)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
287