Mercurial > piecrust2

annotate piecrust/tasks/base.py @ 1144:9f3e702a8a69

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Give unique source specs to each taxonomy or blog archive page. This prevents caching issues in some situations, leading to one tag page reusing the data from a previous tag page.
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 05 Jun 2018 22:05:46 -0700
parents 8af2ea1f5c34
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1114
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import json
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import time
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import logging
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 from piecrust.chefutil import format_timed
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9 TASKS_DIR = '_tasks'
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12 logger = logging.getLogger(__name__)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 class TaskContext:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16 def __init__(self):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 pass
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 class TaskRunner:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 TASK_TYPE = 'undefined'
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 def __init__(self, app):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 self.app = app
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 def runTask(self, task_data, ctx):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 raise NotImplementedError()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 class TaskManager:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 def __init__(self, app, *, time_threshold=1):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32 self.app = app
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 self.time_threshold = time_threshold
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 self._runners = None
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 @property
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 def tasks_dir(self):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 return os.path.join(self.app.root_dir, TASKS_DIR)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 def createTask(self, task_type, task_data):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 from piecrust.pathutil import ensure_dir
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 tasks_dir = self.tasks_dir
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 ensure_dir(tasks_dir)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 new_task = {
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 'type': task_type,
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 'data': task_data}
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 task_id = str(int(time.time()))
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 task_path = os.path.join(tasks_dir, '%s.json' % task_id)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 with open(task_path, 'w', encoding='utf8') as fp:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 json.dump(new_task, fp)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 return task_id
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 def getTasks(self, *, only_task=None):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 max_time = time.time() - self.time_threshold
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56 tasks_dir = self.tasks_dir
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 try:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 task_files = os.listdir(tasks_dir)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 except (IOError, OSError):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60 task_files = []
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62 for tf in task_files:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63 tfname, _ = os.path.splitext(tf)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 if only_task and tfname != only_task:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 continue
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67 tf_path = os.path.join(tasks_dir, tf)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68 task_time = os.path.getmtime(tf_path)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 if task_time >= max_time:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70 logger.debug("Skipping task '%s' because it's too new." % tf)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 continue
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 with open(tf_path, 'r', encoding='utf8') as fp:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 task_data = json.load(fp)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 task_type = task_data.get('task')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 task_payload = task_data.get('data')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 yield (tf_path, task_type, task_payload)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 def runQueue(self, *, only_task=None, clear_queue=True):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81 start_time = time.perf_counter()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83 tasks = list(self.getTasks(only_task=only_task))
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84 for path, task_type, task_data in tasks:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
85 if not task_type:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
86 logger.error("Got task with no type: %s" % path)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
87 continue
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
88
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
89 runner = self._getRunner(task_type)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
90 if runner is None:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
91 logger.error("No task runner for type: %s" % task_type)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
92 continue
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
93
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
94 ctx = TaskContext()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
95 runner.runTask(task_data, ctx)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
96
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
97 if clear_queue:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98 os.remove(path)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
99
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
100 logger.info(format_timed(
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
101 start_time, "Ran %d tasks." % len(tasks)))
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
102
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
103 def _getRunner(self, task_type):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
104 if self._runners is None:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
105 self._runners = {}
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
106 for r in self.app.plugin_loader.getTaskRunners():
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
107 self._runners[r.TASK_TYPE] = r(self.app)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
108
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
109 return self._runners.get(task_type)