Mercurial > piecrust2

annotate piecrust/importing/piecrust.py @ 1088:c5f1936e9e89

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
data: Fix recently introduced bug with blog data provider.
author Ludovic Chabant <ludovic@chabant.com>
date Thu, 15 Feb 2018 23:46:37 -0800
parents 8a96ff3f2b01
children 727110ea112a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
62
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import re
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import shutil
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 import logging
64
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
6 import yaml
62
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7 from piecrust.importing.base import FileWalkingImporter
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10 logger = logging.getLogger(__name__)
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13 class PieCrust1Importer(FileWalkingImporter):
297
2823ea40cfac import: Put importer metadata on the class, and allow return values.
Ludovic Chabant <ludovic@chabant.com>
parents: 291
diff changeset
14 name = 'piecrust1'
2823ea40cfac import: Put importer metadata on the class, and allow return values.
Ludovic Chabant <ludovic@chabant.com>
parents: 291
diff changeset
15 description = "Imports content from a PieCrust 1 website."
2823ea40cfac import: Put importer metadata on the class, and allow return values.
Ludovic Chabant <ludovic@chabant.com>
parents: 291
diff changeset
16 requires_website = False
62
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18 def setupParser(self, parser, app):
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19 super(PieCrust1Importer, self).setupParser(parser, app)
63
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
20 parser.add_argument('root_dir', nargs='?',
62
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21 help="The root directory of the PieCrust 1 website.")
63
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
22 parser.add_argument('--upgrade', action='store_true',
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
23 help="Upgrade the current website in place.")
62
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25 def importWebsite(self, app, args):
540
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
26 if args.root_dir and args.upgrade:
63
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
27 raise Exception("Can't specifiy both a root directory and `--upgrade`.")
540
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
28 if args.root_dir is None and not args.upgrade:
63
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
29 raise Exception("Need to specify either a root directory or `--upgrade`.")
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
30
540
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
31 if app.root_dir is None and not args.upgrade:
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
32 raise Exception("Need to run the import from inside a PieCrust 2 "
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
33 "website. Use `--upgrade` to upgrade from inside "
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
34 "a PieCrust 1 website.")
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
35 if app.root_dir is not None and args.upgrade:
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
36 raise Exception("Already in a PieCrust 2 website. Specify the "
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
37 "PieCrust 1 website to import from.")
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
38
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
39 src_root_dir = os.getcwd() if args.upgrade else args.root_dir
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
40 out_root_dir = src_root_dir if args.upgrade else app.root_dir
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
41 logger.debug("Importing PieCrust 1 site from: %s" % src_root_dir)
62
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 exclude = args.exclude or []
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 exclude += ['_cache', '_counter']
540
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
44 self._startWalk(src_root_dir, exclude, out_root_dir, args.upgrade)
63
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
45 if args.upgrade:
540
233234e02816 import: Fix the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 297
diff changeset
46 self._cleanEmptyDirectories(src_root_dir)
62
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 logger.info("The PieCrust website was successfully imported.")
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48
63
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
49 def _importFile(self, full_fn, rel_fn, out_root_dir, is_move):
62
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 logger.debug("- %s" % rel_fn)
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 dest_path = rel_fn
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 convert_func = None
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 if rel_fn.replace('\\', '/') == '_content/config.yml':
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 dest_path = 'config.yml'
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 convert_func = self.convertConfig
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56 elif rel_fn.startswith('_content'):
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 dest_path = rel_fn[len('_content/'):]
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 fn_dirname = os.path.dirname(rel_fn)
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 if not fn_dirname.endswith('-assets'):
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60 convert_func = self.convertPage
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61 else:
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62 dest_path = 'assets/' + rel_fn
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 logger.debug(" %s -> %s" % (rel_fn, dest_path))
63
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
65 full_dest_path = os.path.join(out_root_dir, dest_path)
62
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66 os.makedirs(os.path.dirname(full_dest_path), 0o755, True)
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67 if convert_func is None:
63
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
68 if is_move:
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
69 shutil.move(full_fn, full_dest_path)
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
70 else:
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
71 shutil.copy2(full_fn, full_dest_path)
62
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 else:
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 with open(full_fn, 'r', encoding='utf8') as fp:
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74 content = fp.read()
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 converted_content = convert_func(content)
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 with open(full_dest_path, 'w', encoding='utf8') as fp:
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 fp.write(converted_content)
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 if converted_content != content:
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 logger.warning("'%s' has been modified. The original version "
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 "has been kept for reference." % rel_fn)
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81 shutil.copy2(full_fn, full_dest_path + '.orig')
63
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
82 if is_move:
28958565a17b In-place upgrade for PieCrust 1 sites.
Ludovic Chabant <ludovic@chabant.com>
parents: 62
diff changeset
83 os.remove(full_fn)
62
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84
64
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
85 def _cleanEmptyDirectories(self, root_dir):
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
86 for item in os.listdir(root_dir):
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
87 if not os.path.isdir(item):
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
88 continue
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
89
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
90 file_count = 0
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
91 item_path = os.path.join(root_dir, item)
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
92 for _, __, filenames in os.walk(item_path):
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
93 file_count += len(filenames)
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
94 if file_count == 0:
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
95 logger.debug("Deleting empty directory: %s" % item)
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
96 shutil.rmtree(item_path)
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
97
62
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98 def convertConfig(self, content):
64
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
99 config = yaml.load(content)
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
100 sitec = config.setdefault('site', {})
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
101 if 'templates_dirs' in sitec:
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
102 tdc = sitec['templates_dirs']
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
103 cl = len('_content/')
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
104 if isinstance(tdc, str) and re.match(r'^_content[/\\]', tdc):
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
105 sitec['templates_dirs'] = tdc[cl:]
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
106 elif isinstance(tdc, list):
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
107 sitec['templates_dirs'] = list(map(
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
108 lambda d: d[cl:] if re.match(r'^_content[/\\]', d) else d,
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
109 tdc))
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
110
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
111 jinjac = config.setdefault('jinja', {})
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
112 jinjac['twig_compatibility'] = True
9ae3237365eb PieCrust 1 import: clean empty directories and convert some config values.
Ludovic Chabant <ludovic@chabant.com>
parents: 63
diff changeset
113
291
46842f71f31f import: Upgrade more settings for the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 64
diff changeset
114 if 'baker' in config:
46842f71f31f import: Upgrade more settings for the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 64
diff changeset
115 if 'skip_patterns' in config['baker']:
46842f71f31f import: Upgrade more settings for the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 64
diff changeset
116 config['baker']['ignore'] = config['baker']['skip_patterns']
46842f71f31f import: Upgrade more settings for the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 64
diff changeset
117 del config['baker']['skip_patterns']
46842f71f31f import: Upgrade more settings for the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 64
diff changeset
118 if 'force_patterns' in config['baker']:
46842f71f31f import: Upgrade more settings for the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 64
diff changeset
119 config['baker']['force'] = config['baker']['force_patterns']
46842f71f31f import: Upgrade more settings for the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 64
diff changeset
120 del config['baker']['force_patterns']
46842f71f31f import: Upgrade more settings for the PieCrust 1 importer.
Ludovic Chabant <ludovic@chabant.com>
parents: 64
diff changeset
121
542
8a96ff3f2b01 import: Correctly convert unicode characters in site configuration.
Ludovic Chabant <ludovic@chabant.com>
parents: 540
diff changeset
122 content = yaml.dump(config, default_flow_style=False,
8a96ff3f2b01 import: Correctly convert unicode characters in site configuration.
Ludovic Chabant <ludovic@chabant.com>
parents: 540
diff changeset
123 allow_unicode=True)
62
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
124 return content
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
125
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
126 def convertPage(self, content):
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
127 return content
52e4d9a1f917 Simple importer for PieCrust 1 websites.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
128