Mercurial > piecrust2

annotate piecrust/serving/procloop.py @ 854:08e02c2a2a1a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
core: Keep refactoring, this time to prepare for generator sources. - Make a few APIs simpler. - Content pipelines create their own jobs, so that generator sources can keep aborting in `getContents`, but rely on their pipeline to generate pages for baking.
author Ludovic Chabant <ludovic@chabant.com>
date Sun, 04 Jun 2017 23:34:28 -0700
parents c2ea75e37540
children 448710d84121
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import time
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import json
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import queue
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 import logging
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
7 import itertools
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 import threading
663
3ceeca7bb71c themes: Add support for a `--theme` argument to `chef`.
Ludovic Chabant <ludovic@chabant.com>
parents: 570
diff changeset
9 from piecrust import CONFIG_PATH, THEME_CONFIG_PATH
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
10 from piecrust.pipelines.base import (
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
11 PipelineJobCreateContext, PipelineJobRunContext, PipelineJobResult,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
12 PipelineManager)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
13 from piecrust.pipelines.records import (
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
14 MultiRecord, MultiRecordHistory)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 logger = logging.getLogger(__name__)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
19 # This flag is for cancelling all long running requests like SSEs.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
20 server_shutdown = False
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
23 class PipelineStatusServerSentEventProducer(object):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
24 """ The producer for Server-Sent Events (SSE) notifying the front-end
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
25 about useful things like assets having been re-processed in the
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
26 background.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
27 Each has its own queue because the user could have multiple pages
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
28 open, each having to display notifications coming from the server.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
29 """
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
30 def __init__(self, proc_loop):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
31 self._proc_loop = proc_loop
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
32 self._queue = queue.Queue()
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 self._start_time = 0
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
34 self._poll_interval = 0.5
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
35 self._ping_interval = 30
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
36 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
37 self._running = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
38
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
39 def addBuildEvent(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
40 self._queue.put_nowait(item)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 def run(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 logger.debug("Starting pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
44 self._proc_loop.addObserver(self)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 self._start_time = time.time()
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
46 self._running = 1
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 outstr = 'event: ping\ndata: started\n\n'
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
51 while self._running == 1 and not server_shutdown:
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 try:
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
53 # We use a short poll interval (less than a second) because
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
54 # we need to catch `server_shutdown` going `True` as soon as
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
55 # possible to exit this thread when the user hits `CTRL+C`.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
56 data = self._queue.get(True, self._poll_interval)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 except queue.Empty:
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
58 # Not exact timing but close enough.
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
59 self._time_between_pings += self._poll_interval
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
60 if self._time_between_pings >= self._ping_interval:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
61 self._time_between_pings = 0
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
62 logger.debug("Sending ping/heartbeat event.")
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
63 outstr = 'event: ping\ndata: 1\n\n'
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
64 yield bytes(outstr, 'utf8')
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
65 continue
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67 logger.debug("Sending pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
68 outstr = (('event: %s\n' % data['type']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
69 ('id: %s\n' % data['id']) +
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
70 ('data: %s\n\n' % json.dumps(data)))
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
71 self._queue.task_done()
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 yield bytes(outstr, 'utf8')
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 def close(self):
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 logger.debug("Closing pipeline status SSE.")
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
76 self._proc_loop.removeObserver(self)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
77 self._running = 2
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
80 class _AssetProcessingInfo:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
81 def __init__(self, source):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
82 self.source = source
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
83 self.paths = set()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
84 self.last_bake_time = time.time()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
85
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
86
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
87 class ProcessingLoop(threading.Thread):
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
88 def __init__(self, appfactory, out_dir):
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
89 super().__init__(name='pipeline-reloader', daemon=True)
666
81d9c3a3a0b5 internal: Get rid of the whole "sub cache" business.
Ludovic Chabant <ludovic@chabant.com>
parents: 663
diff changeset
90 self.appfactory = appfactory
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
91 self.out_dir = out_dir
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
92 self.last_status_id = 0
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
93 self.interval = 1
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
94 self._app = None
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
95 self._proc_infos = None
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
96 self._last_records = None
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
97 self._last_config_mtime = 0
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
98 self._obs = []
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
99 self._obs_lock = threading.Lock()
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
100 config_name = (
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
101 THEME_CONFIG_PATH if appfactory.theme_site else CONFIG_PATH)
680
c2ea75e37540 serve: Fix some crashes introduced by recent refactor.
Ludovic Chabant <ludovic@chabant.com>
parents: 666
diff changeset
102 self._config_path = os.path.join(appfactory.root_dir, config_name)
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
103
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
104 def addObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
105 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
106 self._obs.append(obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
107
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
108 def removeObserver(self, obs):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
109 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
110 self._obs.remove(obs)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
111
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
112 def run(self):
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
113 self._init()
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
114
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
115 self._last_config_mtime = os.path.getmtime(self._config_path)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
116
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
117 while True:
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
118 cur_config_time = os.path.getmtime(self._config_path)
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
119 if self._last_config_mtime < cur_config_time:
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
120 logger.info("Site configuration changed, reloading pipeline.")
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
121 self._last_config_mtime = cur_config_time
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
122 self._init()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
123 self._runPipelines()
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
124 continue
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
125
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
126 for procinfo in self._proc_infos:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
127 # For each assets folder we try to find the first new or
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
128 # modified file. If any, we just run the pipeline on
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
129 # that source.
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
130 found_new_or_modified = False
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
131 for item in procinfo.source.getAllContents():
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
132 path = item.spec
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
133 if path not in procinfo.paths:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
134 logger.debug("Found new asset: %s" % path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
135 procinfo.paths.add(path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
136 found_new_or_modified = True
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
137 break
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
138 if os.path.getmtime(path) > procinfo.last_bake_time:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
139 logger.debug("Found modified asset: %s" % path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
140 found_new_or_modified = True
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
141 break
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
142 if found_new_or_modified:
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
143 self._runPipeline(procinfo)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
144
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
145 time.sleep(self.interval)
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
146
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
147 def _init(self):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
148 self._app = self.appfactory.create()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
149 self._last_records = MultiRecord()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
150
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
151 self._proc_infos = []
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
152 for src in self._app.sources:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
153 if src.config['pipeline'] != 'asset':
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
154 continue
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
155
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
156 procinfo = _AssetProcessingInfo(src)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
157 self._proc_infos.append(procinfo)
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
158
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
159 # Build the list of initial asset files.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
160 for item in src.getAllContents():
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
161 procinfo.paths.add(item.spec)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
162
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
163 def _runPipelines(self):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
164 record_histories = MultiRecordHistory(MultiRecord(), self._records)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
165 self._ppmngr = PipelineManager(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
166 self._app, self.out_dir, record_histories)
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
167
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
168 # Create the pipelines, but also remember some stuff for what
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
169 # we want to do.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
170 for src in self._app.sources:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
171 if src.config['pipeline'] != 'asset':
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
172 continue
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
173
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
174 ppinfo = self._ppmngr.createPipeline(src)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
175 api = _AssetProcessingInfo()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
176 ppinfo.userdata = api
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
177
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
178 current_records = MultiRecord()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
179 record_histories = MultiRecordHistory(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
180 self._records, current_records)
570
7dabfdd056a1 serve: Fix corner cases where the pipeline doesn't run correctly.
Ludovic Chabant <ludovic@chabant.com>
parents: 565
diff changeset
181
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
182 for ppinfo, procinfo in self._pipelines:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
183 self._runPipeline(ppinfo, procinfo, record_histories)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
184
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
185 status_id = self.last_status_id + 1
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
186 self.last_status_id += 1
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
187
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
188 if self._records.success:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
189 changed = filter(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
190 lambda i: not i.was_collapsed_from_last_run,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
191 self._record.entries)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
192 changed = itertools.chain.from_iterable(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
193 map(lambda i: i.rel_outputs, changed))
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
194 changed = list(changed)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
195 item = {
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
196 'id': status_id,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
197 'type': 'pipeline_success',
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
198 'assets': changed}
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
199
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
200 self._notifyObservers(item)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
201 else:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
202 item = {
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
203 'id': status_id,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
204 'type': 'pipeline_error',
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
205 'assets': []}
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
206 for entry in self._record.entries:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
207 if entry.errors:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
208 asset_item = {
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
209 'path': entry.path,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
210 'errors': list(entry.errors)}
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
211 item['assets'].append(asset_item)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
212
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
213 self._notifyObservers(item)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
214
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
215 def _runPipeline(self, procinfo):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
216 procinfo.last_bake_time = time.time()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
217
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
218 src = procinfo.source
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
219
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
220 current_records = MultiRecord()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
221 record_histories = MultiRecordHistory(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
222 self._last_records, current_records)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
223 ppmngr = PipelineManager(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
224 self._app, self.out_dir, record_histories)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
225 ppinfo = ppmngr.createPipeline(src)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
226
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
227 logger.debug("Running pipeline '%s' on: %s" %
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
228 (ppinfo.pipeline_name, src.name))
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
229
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
230 # Process all items in the source.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
231 pp = ppinfo.pipeline
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
232 cr = ppinfo.record_history.current
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
233 jobctx = PipelineJobCreateContext(src)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
234 for item in src.getAllContents():
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
235 job = pp.createJob(item, jobctx)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
236
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
237 ppres = PipelineJobResult()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
238 ppres.record_entry = pp.createRecordEntry(job)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
239
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
240 runctx = PipelineJobRunContext(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
241 ppinfo.pipeline_ctx, job, record_histories)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
242 try:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
243 pp.run(item, runctx, ppres)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
244 except Exception as e:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
245 ppres.record_entry.errors.append(str(e))
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
246
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
247 if ppres.next_pass_job is not None:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
248 logger.error("The processing loop for the server "
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
249 "doesn't support multi-pass pipelines.")
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
250
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
251 cr.addEntry(ppres.record_entry)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
252 if not ppres.record_entry.success:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
253 cr.success = False
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
254 current_records.success = False
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
255 logger.error("Errors found in %s:" % item.spec)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
256 for e in ppres.record_entry.errors:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
257 logger.error(" " + e)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
258
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
259 # Do all the final stuff.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
260 ppmngr.buildHistoryDiffs()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
261 ppmngr.deleteStaleOutputs()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
262 ppmngr.collapseRecords()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
263 ppmngr.shutdownPipelines()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
264
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
265 # Swap the old record with the next record.
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
266 pr = ppinfo.record_history.previous
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
267 self._last_records.records.remove(pr)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 680
diff changeset
268 self._last_records.records.append(cr)
374
fa3ee8a8ee2d serve: Split the server code in a couple modules inside a `serving` package.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
269
552
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
270 def _notifyObservers(self, item):
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
271 with self._obs_lock:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
272 observers = list(self._obs)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
273 for obs in observers:
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
274 obs.addBuildEvent(item)
9612cfc6455a serve: Rewrite of the Server-Sent Event code for build notifications.
Ludovic Chabant <ludovic@chabant.com>
parents: 374
diff changeset
275