Mercurial > piecrust2

annotate piecrust/commands/builtin/util.py @ 411:e7b865f8f335

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Enable multiprocess baking. Baking is now done by running a worker per CPU, and sending jobs to them. This changes several things across the codebase: * Ability to not cache things related to pages other than the 'main' page (i.e. the page at the bottom of the execution stack). * Decouple the baking process from the bake records, so only the main process keeps track (and modifies) the bake record. * Remove the need for 'batch page getters' and loading a page directly from the page factories. There are various smaller changes too included here, including support for scope performance timers that are saved with the bake record and can be printed out to the console. Yes I got carried away. For testing, the in-memory 'mock' file-system doesn't work anymore, since we're spawning processes, so this is replaced by a 'tmpfs' file-system which is saved in temporary files on disk and deleted after tests have run.
author Ludovic Chabant <ludovic@chabant.com>
date Fri, 12 Jun 2015 17:09:19 -0700
parents c2ca72fb7f0b
children 3ceeca7bb71c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
3 import shutil
1
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import codecs
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import logging
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6 import yaml
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7 from piecrust.app import CONFIG_PATH
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 from piecrust.commands.base import ChefCommand
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11 logger = logging.getLogger(__name__)
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 class InitCommand(ChefCommand):
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 def __init__(self):
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16 super(InitCommand, self).__init__()
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17 self.name = 'init'
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 self.description = "Creates a new empty PieCrust website."
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19 self.requires_website = False
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
21 def setupParser(self, parser, app):
1
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22 parser.add_argument('destination',
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 help="The destination directory in which to create the website.")
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 def run(self, ctx):
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 destination = ctx.args.destination
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 if destination is None:
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 destination = os.getcwd()
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 if not os.path.isdir(destination):
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
31 os.makedirs(destination, 0o755)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
32
1
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 config_path = os.path.join(destination, CONFIG_PATH)
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 if not os.path.isdir(os.path.dirname(config_path)):
5
474c9882decf Upgrade to Python 3.
Ludovic Chabant <ludovic@chabant.com>
parents: 3
diff changeset
35 os.makedirs(os.path.dirname(config_path), 0o755)
1
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 config_text = yaml.dump({
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 'site': {
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 'title': "My New Website",
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 'description': "A website recently generated with PieCrust",
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 'pretty_urls': True
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 },
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 'smartypants': {
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 'enable': True
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 }
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 },
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 default_flow_style=False)
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 with codecs.open(config_path, 'w', 'utf-8') as fp:
aaa8fb7c8918 Re-arranged modules to reduce dependencies to builtin stuff.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 fp.write(config_text)
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
50
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
51
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
52 class PurgeCommand(ChefCommand):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
53 def __init__(self):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
54 super(PurgeCommand, self).__init__()
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
55 self.name = 'purge'
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
56 self.description = "Purges the website's cache."
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
57
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
58 def setupParser(self, parser, app):
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
59 pass
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
60
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
61 def run(self, ctx):
371
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents: 304
diff changeset
62 cache_dir = ctx.app.sub_cache_dir
c2ca72fb7f0b caching: Use separate caches for config variants and other contexts.
Ludovic Chabant <ludovic@chabant.com>
parents: 304
diff changeset
63 if cache_dir and os.path.isdir(cache_dir):
3
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
64 logger.info("Purging cache: %s" % cache_dir)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
65 shutil.rmtree(cache_dir)
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
66
f485ba500df3 Gigantic change to basically make PieCrust 2 vaguely functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 1
diff changeset
67
60
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
68 class ImportCommand(ChefCommand):
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
69 def __init__(self):
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
70 super(ImportCommand, self).__init__()
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
71 self.name = 'import'
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
72 self.description = "Imports content from another CMS into PieCrust."
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
73
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
74 def setupParser(self, parser, app):
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
75 subparsers = parser.add_subparsers()
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
76 for i in app.plugin_loader.getImporters():
297
2823ea40cfac import: Put importer metadata on the class, and allow return values.
Ludovic Chabant <ludovic@chabant.com>
parents: 100
diff changeset
77 if not i.__class__.name:
2823ea40cfac import: Put importer metadata on the class, and allow return values.
Ludovic Chabant <ludovic@chabant.com>
parents: 100
diff changeset
78 raise Exception("Importer '%s' has no name set." % type(i))
60
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
79 p = subparsers.add_parser(i.name, help=i.description)
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
80 i.setupParser(p, app)
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
81 p.set_defaults(sub_func=i.checkedImportWebsite)
63
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 60
diff changeset
82 p.set_defaults(sub_requires_website=i.requires_website)
60
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
83
63
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 60
diff changeset
84 def checkedRun(self, ctx):
304
34ef6a2a0c97 import: Show help if no sub-command was specified.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
85 if not hasattr(ctx.args, 'sub_func'):
34ef6a2a0c97 import: Show help if no sub-command was specified.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
86 ctx.parser.parse_args(['import', '--help'])
34ef6a2a0c97 import: Show help if no sub-command was specified.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
87 return
60
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
88 ctx.args.sub_func(ctx)
6e60e0fef2be Add `import` command, Jekyll importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 51
diff changeset
89