piecrust2: piecrust/importing/jekyll.py annotate

annotate piecrust/importing/jekyll.py @ 1188:a7c43131d871

bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(

author	Ludovic Chabant <ludovic@chabant.com>
date	Tue, 15 Jun 2021 22:36:23 -0700
parents	727110ea112a
children

rev	line source
60 6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import os
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os.path
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import re
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	4 import shutil
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import yaml
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 import logging
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	7 from piecrust.configuration import parse_config_header
62 52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	8 from piecrust.importing.base import FileWalkingImporter
60 6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	9 from piecrust.uriutil import multi_replace
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	10
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	11
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12 logger = logging.getLogger(__name__)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	14
62 52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	15 class JekyllImporter(FileWalkingImporter):
297 2823ea40cfac import: Put importer metadata on the class, and allow return values. Ludovic Chabant <ludovic@chabant.com> parents: 62 diff changeset	16 name = 'jekyll'
2823ea40cfac import: Put importer metadata on the class, and allow return values. Ludovic Chabant <ludovic@chabant.com> parents: 62 diff changeset	17 description = "Imports content from a Jekyll or Octopress blog."
60 6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19 def setupParser(self, parser, app):
62 52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	20 super(JekyllImporter, self).setupParser(parser, app)
60 6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21 parser.add_argument('root_dir',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22 help="The root directory of the Jekyll or Octopress website.")
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	23
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24 def importWebsite(self, app, args):
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	25 logger.debug("Importing Jekyll site from: %s" % args.root_dir)
62 52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	26 self._startWalk(args.root_dir, args.exclude, app)
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	27 logger.info("The Jekyll website was successfully imported.")
60 6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28
62 52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	29 def _importFile(self, full_fn, rel_fn, app):
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	30 logger.debug("- %s" % rel_fn)
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	31 if rel_fn == '_config.yml':
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	32 self.convertConfig(app, full_fn)
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	33 elif rel_fn.startswith('_layouts'):
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	34 self.convertLayout(app, full_fn, rel_fn[len('_layouts/'):])
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	35 elif rel_fn.startswith('_includes'):
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	36 self.convertInclude(app, full_fn, rel_fn[len('_includes/'):])
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	37 elif rel_fn.startswith('_posts'):
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	38 self.convertPost(app, full_fn, rel_fn[len('_posts/'):])
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	39 else:
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	40 with open(full_fn, 'rb') as fp:
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	41 firstline = fp.read(3)
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	42 if firstline == '---':
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	43 self.convertPage(app, full_fn, rel_fn)
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	44 else:
52e4d9a1f917 Simple importer for PieCrust 1 websites. Ludovic Chabant <ludovic@chabant.com> parents: 60 diff changeset	45 self.convertStatic(app, full_fn, rel_fn)
60 6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	46
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	47 def convertConfig(self, app, src_path):
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	48 logger.debug(" Converting configuration file.")
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49 with open(src_path, 'r', encoding='utf8') as fp:
1164 727110ea112a core: Remove more YAML deprecation warnings. Ludovic Chabant <ludovic@chabant.com> parents: 355 diff changeset	50 config = yaml.safe_load(fp)
60 6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	51
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	52 if 'site' not in config:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	53 config['site'] = {}
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	54 config['site']['related_posts'] = []
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	55 config['site']['posts_fs'] = 'flat'
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	56 config['site']['templates_dirs'] = ['includes', 'layouts']
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	57 config['site']['tag_url'] = 'tags/%tag%'
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	58 if 'permalink' in config:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 permalink = config['permalink']
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	60 if permalink == 'date':
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	61 permalink = '/:categories/:year/:month/:day/:title.html'
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62 elif permalink == 'pretty':
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63 permalink = '/:categories/:year/:month/:day/:title/'
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64 elif permalink == 'none':
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	65 permalink = '/:categories/:title.html'
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	66
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	67 # TODO: handle `:categories` token.
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 post_url = multi_replace(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69 permalink,
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	70 {':year': '%year%', ':month': '%month%', ':day': '%day%',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71 ':title': '%slug%', ':categories': ''})
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	72 post_url = post_url.replace('//', '/').strip('/')
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	73 config['site']['post_url'] = post_url
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	74 if 'exclude' in config:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75 if 'baker' not in config:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76 config['baker'] = {}
355 28b021be74eb import: Use the proper baker setting in the Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: 297 diff changeset	77 config['baker']['ignore'] = list(map(
60 6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	78 lambda i: '^/_%s/' % re.escape(i)))
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	79 if 'jinja' not in config:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80 config['jinja'] = {}
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	81 config['jinja']['auto_escape'] = False
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	82 if 'markdown' in config:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	83 if not isinstance(config['markdown'], dict):
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	84 logger.warning("Discarding markdown setting: %s" %
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	85 config['markdown'])
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	86 del config['markdown']
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	87
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	88 with open(os.path.join(app.root_dir, 'config.yml'), 'w') as fp:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	89 yaml.dump(config, stream=fp)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	90
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	91 def convertPage(self, app, path, rel_path):
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	92 logger.debug(" Converting page: %s" % rel_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	93 is_index = False
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	94 is_static = False
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	95 _, ext = os.path.splitext(rel_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	96 if re.search(r'^index\.(html?\|textile\|markdown)$', rel_path):
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	97 out_path = os.path.join(app.root_dir, 'pages', '_index' + ext)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	98 is_index = True
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	99 else:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	100 out_path = os.path.join(app.root_dir, 'pages', rel_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	101
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	102 if ext not in ['htm', 'html', 'textile', 'markdown']:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	103 # There could be static files (SCSS or Less files) that look like
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	104 # pages because they have a YAML front matter.
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	105 is_static = True
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	106 out_path = os.path.join(app.root_dir, 'assets', rel_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	107
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	108 if is_static:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	109 logger.debug(" Actually a static file... forwarding converstion.")
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	110 self.convertStatic(app, path, rel_path, True)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	111 return
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	112
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	113 self._doConvertPage(app, path, out_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	114 if is_index:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	115 shutil.copy2(out_path, os.path.join(app.root_dir, 'pages', '_tag.%s' % ext))
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	116
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	117 def convertPost(self, app, path, rel_path):
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	118 logger.debug(" Converting post: %s" % rel_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	119 out_path = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	120 r'(\d{4}\-\d{2}\-\d{2})\-(.*)$',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	121 r'\1_\2',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	122 rel_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	123 out_path = os.path.join(app.root_dir, 'posts', out_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	124 self._doConvertPage(app, path, out_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	125
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	126 def convertLayout(self, app, path, rel_path):
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	127 logger.debug(" Converting layout: %s" % rel_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	128 out_path = os.path.join(app.root_dir, 'layouts', rel_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	129 self._doConvertPage(app, path, out_path, True)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	130
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	131 def convertInclude(self, app, path, rel_path):
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	132 logger.debug(" Converting include: %s" % rel_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	133 out_path = os.path.join(app.root_dir, 'includes', rel_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	134 self._doConvertPage(app, path, out_path, True)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	135
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	136 def convertStatic(self, app, path, rel_path, strip_header=False):
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	137 logger.debug(" Converting static: %s" % rel_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	138 out_path = os.path.join(app.root_dir, 'assets', rel_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	139 logger.debug(" %s -> %s" % (path, out_path))
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	140 os.makedirs(os.path.dirname(out_path), 0o755, True)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	141
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	142 if strip_header:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	143 with open(path, 'r', encoding='utf8') as fp:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	144 content = fp.write()
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	145 config, offset = parse_config_header(content)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	146 content = content[offset:]
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	147 with open(out_path, 'w', encoding='utf8') as fp:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	148 fp.write(content)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	149 return
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	151 shutil.copy2(path, out_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	152
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	153 def _doConvertPage(self, app, path, out_path, is_template=False):
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	154 logger.debug(" %s -> %s" % (path, out_path))
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	155 os.makedirs(os.path.dirname(out_path), 0o755, True)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	156
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	157 with open(path, 'r', encoding='utf8') as fp:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	158 contents = fp.read()
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	159
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	160 config, offset = parse_config_header(contents)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	161 text = contents[offset:]
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	162 text_before = text
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	163
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	164 wrap_content_tag = True
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	165
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	166 if is_template:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	167 if 'layout' in config:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	168 # Liquid doesn't support template inheritance but
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	169 # Jinja does.
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	170 text = ("{%% extends '%s.html' %%}\n\n"
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	171 "{%% block jekyllcontent %%}\n"
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	172 "%s\n"
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	173 "{%% endblock %%}\n" % (config['layout'], text))
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	174 wrap_content_tag = False
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	175 else:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	176 if 'layout' in config:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	177 if config['layout'] == 'nil':
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	178 config['layout'] = 'none'
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	179
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	180 # Convert the template stuff we can:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	181 # - content tag may have to be wrapped in a `jekyllcontent`
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	182 # because Jekyll uses implicit layout inheritance
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	183 # placements.
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	184 if wrap_content_tag:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	185 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	186 r'{{\scontent\s}}',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	187 r'{% block jekyllcontent %}{{ content }}{% endblock %}',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	188 text)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	189 # - list of posts
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	190 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	191 '(?<=\{%\|\{\{)([^\}]*)site.posts',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	192 '\\1blog.posts',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	193 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	194 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	195 '(?<=\{%\|\{\{)([^\}]*)paginator.posts',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	196 '\\1pagination.posts',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	197 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	198 # - list of categories or tags
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	199 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	200 '(?<=\{%\|\{\{)([^\}]*)site.categories',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	201 '\\1blog.categories',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	202 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	203 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	204 '(?<=\{%\|\{\{)([^\}]*)site.tags',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	205 '\\1blog.tags',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	206 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	207 # - list of related posts
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	208 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	209 '(?<=\{%\|\{\{)(?<!%\})site.related_posts',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	210 '\\1pagination.related_posts',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	211 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	212 # - enumeration limits
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	213 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	214 '{%\sfor\s+([^}]+)\s+limit\:\s(\d+)',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	215 '{% for \\1[:\\2]',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	216 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	217 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	218 '{%\sfor\s+([^}]+)\s+offset\:\s(\d+)',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	219 '{% for \\1[\\2:]',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	220 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	221 # - code highlighting
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	222 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	223 '{%\shighlight\s+([\w\d]+)\s%}',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	224 '{% geshi \'\\1\' %}',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	225 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	226 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	227 '{%\sendhighlight\s%}',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	228 '{% endgeshi %}',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	229 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	230 # - unless tag
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	231 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	232 '{%\sunless\s+([^}]+)\s%}',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	233 '{% if not \\1 %}',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	234 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	235 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	236 '{%\sendunless\s%}',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	237 '{% endif %}',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	238 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	239 # - variable assignment
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	240 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	241 '\{%\s*assign\s+',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	242 '{% set ',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	243 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	244 # - include tag
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	245 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	246 '\{%\sinclude\s+([\w\d\.\-_]+)\s%}',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	247 '{% include "\\1" %}',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	248 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	249 # - truncate filter
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	250 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	251 '\\|\struncate\:\s(\d+)',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	252 '\|truncate(\\1)',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	253 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	254 # - date filter
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	255 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	256 '\\|\sdate\:\s"([^"]+)"',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	257 '\|date("\\1")',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	258 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	259 # - some filters we don't need
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	260 text = re.sub(
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	261 '\\|\s*date_to_string',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	262 '',
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	263 text);
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	264
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	265 if text != text_before:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	266 # We changed the text, so create a backup.
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	267 shutil.copy2(path, '%s.orig' % out_path)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	268
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	269 with open(out_path, 'w', encoding='utf8') as fp:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	270 if not is_template:
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	271 fp.write("---\n")
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	272 fp.write(yaml.dump(config))
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	273 fp.write("---\n")
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	274 fp.write(text)
6e60e0fef2be Add `import` command, Jekyll importer. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	275

Mercurial > piecrust2

annotate piecrust/importing/jekyll.py @ 1188:a7c43131d871