Mercurial > piecrust2

annotate piecrust/tasks/base.py @ 1188:a7c43131d871

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 15 Jun 2021 22:36:23 -0700
parents 8af2ea1f5c34
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1114
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import json
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import time
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import logging
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 from piecrust.chefutil import format_timed
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 TASKS_DIR = '_tasks'
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12 logger = logging.getLogger(__name__)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 class TaskContext:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16 def __init__(self):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 pass
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 class TaskRunner:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 TASK_TYPE = 'undefined'
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 def __init__(self, app):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 self.app = app
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 def runTask(self, task_data, ctx):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 raise NotImplementedError()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 class TaskManager:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 def __init__(self, app, *, time_threshold=1):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 self.app = app
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 self.time_threshold = time_threshold
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 self._runners = None
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 @property
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 def tasks_dir(self):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 return os.path.join(self.app.root_dir, TASKS_DIR)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 def createTask(self, task_type, task_data):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 from piecrust.pathutil import ensure_dir
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 tasks_dir = self.tasks_dir
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 ensure_dir(tasks_dir)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 new_task = {
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 'type': task_type,
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 'data': task_data}
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 task_id = str(int(time.time()))
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 task_path = os.path.join(tasks_dir, '%s.json' % task_id)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 with open(task_path, 'w', encoding='utf8') as fp:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 json.dump(new_task, fp)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 return task_id
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 def getTasks(self, *, only_task=None):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 max_time = time.time() - self.time_threshold
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56 tasks_dir = self.tasks_dir
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 try:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 task_files = os.listdir(tasks_dir)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 except (IOError, OSError):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60 task_files = []
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62 for tf in task_files:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63 tfname, _ = os.path.splitext(tf)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 if only_task and tfname != only_task:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 continue
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67 tf_path = os.path.join(tasks_dir, tf)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68 task_time = os.path.getmtime(tf_path)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 if task_time >= max_time:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70 logger.debug("Skipping task '%s' because it's too new." % tf)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 continue
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 with open(tf_path, 'r', encoding='utf8') as fp:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 task_data = json.load(fp)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 task_type = task_data.get('task')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 task_payload = task_data.get('data')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 yield (tf_path, task_type, task_payload)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 def runQueue(self, *, only_task=None, clear_queue=True):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81 start_time = time.perf_counter()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83 tasks = list(self.getTasks(only_task=only_task))
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84 for path, task_type, task_data in tasks:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
85 if not task_type:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
86 logger.error("Got task with no type: %s" % path)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
87 continue
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
88
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
89 runner = self._getRunner(task_type)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
90 if runner is None:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
91 logger.error("No task runner for type: %s" % task_type)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
92 continue
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
93
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
94 ctx = TaskContext()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
95 runner.runTask(task_data, ctx)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
96
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
97 if clear_queue:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98 os.remove(path)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
99
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
100 logger.info(format_timed(
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
101 start_time, "Ran %d tasks." % len(tasks)))
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
102
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
103 def _getRunner(self, task_type):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
104 if self._runners is None:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
105 self._runners = {}
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
106 for r in self.app.plugin_loader.getTaskRunners():
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
107 self._runners[r.TASK_TYPE] = r(self.app)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
108
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
109 return self._runners.get(task_type)