Mercurial > piecrust2

annotate piecrust/serving/procloop.py @ 870:48d25fd68b8d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
assets: Fix bug in assetor.
author Ludovic Chabant <ludovic@chabant.com>
date Mon, 12 Jun 2017 22:30:06 -0700
parents c71472e6537f
children d1095774bfcf
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import time
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import json
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import queue
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 import logging
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
7 import itertools
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 import threading
663
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
9 from piecrust import CONFIG_PATH, THEME_CONFIG_PATH
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
10 from piecrust.pipelines.base import (
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
11 PipelineJobCreateContext, PipelineJobRunContext, PipelineJobResult,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
12 PipelineManager)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
13 from piecrust.pipelines.records import (
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
14 MultiRecord, MultiRecordHistory)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 logger = logging.getLogger(__name__)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
19 # This flag is for cancelling all long running requests like SSEs.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
20 server_shutdown = False
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
23 class PipelineStatusServerSentEventProducer(object):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
24 """ The producer for Server-Sent Events (SSE) notifying the front-end
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
25 about useful things like assets having been re-processed in the
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
26 background.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
27 Each has its own queue because the user could have multiple pages
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
28 open, each having to display notifications coming from the server.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
29 """
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
30 def __init__(self, proc_loop):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
31 self._proc_loop = proc_loop
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
32 self._queue = queue.Queue()
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 self._start_time = 0
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
34 self._poll_interval = 0.5
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
35 self._ping_interval = 30
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
36 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
37 self._running = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
38
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
39 def addBuildEvent(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
40 self._queue.put_nowait(item)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 def run(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 logger.debug("Starting pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
44 self._proc_loop.addObserver(self)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 self._start_time = time.time()
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
46 self._running = 1
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 outstr = 'event: ping\ndata: started\n\n'
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
51 while self._running == 1 and not server_shutdown:
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 try:
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
53 # We use a short poll interval (less than a second) because
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
54 # we need to catch `server_shutdown` going `True` as soon as
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
55 # possible to exit this thread when the user hits `CTRL+C`.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
56 data = self._queue.get(True, self._poll_interval)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 except queue.Empty:
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
58 # Not exact timing but close enough.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
59 self._time_between_pings += self._poll_interval
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
60 if self._time_between_pings >= self._ping_interval:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
61 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
62 logger.debug("Sending ping/heartbeat event.")
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
63 outstr = 'event: ping\ndata: 1\n\n'
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
64 yield bytes(outstr, 'utf8')
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
65 continue
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67 logger.debug("Sending pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
68 outstr = (('event: %s\n' % data['type']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
69 ('id: %s\n' % data['id']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
70 ('data: %s\n\n' % json.dumps(data)))
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
71 self._queue.task_done()
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 def close(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 logger.debug("Closing pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
76 self._proc_loop.removeObserver(self)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
77 self._running = 2
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
80 class _AssetProcessingInfo:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
81 def __init__(self, source):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
82 self.source = source
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
83 self.paths = set()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
84 self.last_bake_time = time.time()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
85
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
86
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
87 class ProcessingLoop(threading.Thread):
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
88 def __init__(self, appfactory, out_dir):
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
89 super().__init__(name='pipeline-reloader', daemon=True)
666
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business.
Ludovic Chabant <ludovic@chabant.com>
parents: 663
diff changeset
90 self.appfactory = appfactory
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
91 self.out_dir = out_dir
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
92 self.last_status_id = 0
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
93 self.interval = 1
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
94 self._app = None
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
95 self._proc_infos = None
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
96 self._last_records = None
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
97 self._last_config_mtime = 0
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
98 self._obs = []
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
99 self._obs_lock = threading.Lock()
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
100 config_name = (
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
101 THEME_CONFIG_PATH if appfactory.theme_site else CONFIG_PATH)
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
102 self._config_path = os.path.join(appfactory.root_dir, config_name)
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
103
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
104 def addObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
105 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
106 self._obs.append(obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
107
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
108 def removeObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
109 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
110 self._obs.remove(obs)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
111
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
112 def run(self):
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
113 logger.debug("Initializing processing loop with output: %s" %
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
114 self.out_dir)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
115 try:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
116 self._init()
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
117 except Exception as ex:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
118 logger.error("Error initializing processing loop:")
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
119 logger.exception(ex)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
120 return
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
121
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
122 logger.debug("Doing initial processing loop bake...")
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
123 self._runPipelines()
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
124
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
125 logger.debug("Running processing loop...")
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
126 self._last_config_mtime = os.path.getmtime(self._config_path)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
127
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
128 while True:
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
129 cur_config_time = os.path.getmtime(self._config_path)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
130 if self._last_config_mtime < cur_config_time:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
131 logger.info("Site configuration changed, reloading pipeline.")
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
132 self._last_config_mtime = cur_config_time
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
133 self._init()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
134 self._runPipelines()
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
135 continue
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
136
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
137 for procinfo in self._proc_infos.values():
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
138 # For each assets folder we try to find the first new or
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
139 # modified file. If any, we just run the pipeline on
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
140 # that source.
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
141 found_new_or_modified = False
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
142 for item in procinfo.source.getAllContents():
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
143 path = item.spec
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
144 if path not in procinfo.paths:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
145 logger.debug("Found new asset: %s" % path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
146 procinfo.paths.add(path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
147 found_new_or_modified = True
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
148 break
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
149 if os.path.getmtime(path) > procinfo.last_bake_time:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
150 logger.debug("Found modified asset: %s" % path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
151 found_new_or_modified = True
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
152 break
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
153 if found_new_or_modified:
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
154 self._runPipelines(procinfo.source)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
155
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
156 time.sleep(self.interval)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
157
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
158 def _init(self):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
159 self._app = self.appfactory.create()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
160 self._last_records = MultiRecord()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
161
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
162 self._proc_infos = {}
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
163 for src in self._app.sources:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
164 if src.config['pipeline'] != 'asset':
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
165 continue
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
166
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
167 procinfo = _AssetProcessingInfo(src)
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
168 self._proc_infos[src.name] = procinfo
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
169
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
170 # Build the list of initial asset files.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
171 for item in src.getAllContents():
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
172 procinfo.paths.add(item.spec)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
173
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
174 def _runPipelines(self, only_for_source=None):
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
175 current_records = MultiRecord()
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
176 record_histories = MultiRecordHistory(
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
177 self._last_records, current_records)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
178 ppmngr = PipelineManager(
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
179 self._app, self.out_dir, record_histories)
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
180
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
181 # Create the pipelines, but also remember some stuff for what
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
182 # we want to do.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
183 for src in self._app.sources:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
184 if src.config['pipeline'] != 'asset':
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
185 continue
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
186 if only_for_source is not None and src != only_for_source:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
187 continue
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
188
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
189 ppmngr.createPipeline(src)
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
190
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
191 for ppinfo in ppmngr.getPipelines():
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
192 self._runPipeline(ppmngr, ppinfo)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
193
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
194 self.last_status_id += 1
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
195
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
196 if self._last_records.success:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
197 for rec in self._last_records.records:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
198 changed = filter(
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
199 lambda i: not i.was_collapsed_from_last_run,
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
200 rec.getEntries())
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
201 changed = itertools.chain.from_iterable(
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
202 map(lambda i: i.out_paths, changed))
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
203 changed = list(changed)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
204 item = {
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
205 'id': self.last_status_id,
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
206 'type': 'pipeline_success',
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
207 'assets': changed}
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
208
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
209 self._notifyObservers(item)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
210 else:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
211 item = {
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
212 'id': self.last_status_id,
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
213 'type': 'pipeline_error',
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
214 'assets': []}
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
215 for rec in self._last_records.records:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
216 for entry in rec.getEntries():
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
217 if entry.errors:
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
218 asset_item = {
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
219 'path': entry.item_spec,
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
220 'errors': list(entry.errors)}
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
221 item['assets'].append(asset_item)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
222
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
223 self._notifyObservers(item)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
224
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
225 def _runPipeline(self, ppmngr, ppinfo):
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
226 src = ppinfo.source
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
227 logger.debug("Running pipeline '%s' on: %s" %
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
228 (ppinfo.pipeline_name, src.name))
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
229
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
230 # Set the time.
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
231 procinfo = self._proc_infos[src.name]
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
232 procinfo.last_bake_time = time.time()
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
233
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
234 # Process all items in the source.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
235 pp = ppinfo.pipeline
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
236 cr = ppinfo.record_history.current
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
237 record_histories = ppmngr.record_histories
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
238 current_records = record_histories.current
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
239 jobctx = PipelineJobCreateContext(0, record_histories)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
240 jobs = pp.createJobs(jobctx)
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
241 for job in jobs:
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
242 ppres = PipelineJobResult()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
243 ppres.record_entry = pp.createRecordEntry(job)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
244
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
245 runctx = PipelineJobRunContext(
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
246 job, pp, record_histories)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
247 try:
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
248 pp.run(job, runctx, ppres)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
249 except Exception as e:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
250 ppres.record_entry.errors.append(str(e))
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
251
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
252 if ppres.next_pass_job is not None:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
253 logger.error("The processing loop for the server "
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
254 "doesn't support multi-pass pipelines.")
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
255
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
256 cr.addEntry(ppres.record_entry)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
257 if not ppres.record_entry.success:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
258 cr.success = False
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
259 current_records.success = False
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
260 logger.error("Errors found in %s:" % job.content_item.spec)
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
261 for e in ppres.record_entry.errors:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
262 logger.error(" " + e)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
263
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
264 # Do all the final stuff.
855
448710d84121 refactor: Get the taxonomy support back to a functional state.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
265 ppmngr.postJobRun()
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
266 ppmngr.deleteStaleOutputs()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
267 ppmngr.collapseRecords()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
268 ppmngr.shutdownPipelines()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
269
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
270 # Swap the old record with the next record.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
271 pr = ppinfo.record_history.previous
860
c71472e6537f refactor: Get the processing loop in the server functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 855
diff changeset
272 logger.debug("Swapping record '%s' with '%s'." % (pr.name, cr.name))
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
273 self._last_records.records.remove(pr)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
274 self._last_records.records.append(cr)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
275
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
276 def _notifyObservers(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
277 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
278 observers = list(self._obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
279 for obs in observers:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
280 obs.addBuildEvent(item)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
281