Mercurial > piecrust2

annotate piecrust/tasks/mentions.py @ 1168:10520472cc73

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
routing: Fix breakages with routing on some versions of Python. Finally figured what happened with change 6baa94da8b16 (this is a Mercurial hash by the way if you're looking at the Git mirror). Between Python 3.6 and 3.7 there was a change where the percent sign ('%') went from being escaped by `re.escape` to _not_ being escaped. So now we need to use different regex patterns dependin on the Python version, yay.
author Ludovic Chabant <ludovic@chabant.com>
date Fri, 04 Oct 2019 11:13:33 -0700
parents 8af2ea1f5c34
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1114
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import json
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import logging
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 from piecrust.tasks.base import TaskRunner
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 logger = logging.getLogger(__name__)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11 class InvalidMentionTargetError(Exception):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12 pass
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 class SourceDoesntLinkToTargetError(Exception):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16 pass
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19 class DuplicateMentionError(Exception):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 pass
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 class MentionTaskRunner(TaskRunner):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 TASK_TYPE = 'mention'
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 def runTask(self, data, ctx):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 import json
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 import requests
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29 from bs4 import BeautifulSoup
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 from piecrust.app import PieCrustFactory
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 from piecrust.serving.util import get_requested_page
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 src_url = data['source']
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 tgt_url = data['target']
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 # Find if we have a page at the target URL. To do that we need to spin
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 # up a PieCrust app that knows how the website works. Because the
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 # website might have been baked with custom settings (usually the site
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 # root URL) there's a good chance we need to apply some variants, which
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 # the user can specify in the config.
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 pcappfac = PieCrustFactory(self.app.root_dir,
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 cache_key='webmention')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 wmcfg = self.app.config.get('webmention')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 if wmcfg.get('config_variant'):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 pcappfac.config_variants = [wmcfg.get('config_variant')]
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 if wmcfg.get('config_variants'):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 pcappfac.config_variants = list(wmcfg.get('config_variants'))
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 if wmcfg.get('config_values'):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 pcappfac.config_values = list(wmcfg.get('config_values').items())
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 pcapp = pcappfac.create()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 logger.debug("Locating page: %s" % tgt_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 try:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 req_page = get_requested_page(pcapp, tgt_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 if req_page.page is None:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 raise InvalidMentionTargetError()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56 except Exception as ex:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 logger.error("Can't check webmention target page: %s" % tgt_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 logger.exception(ex)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 raise InvalidMentionTargetError()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61 # Grab the source URL's contents and see if anything references the
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62 # target (ours) URL.
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63 logger.debug("Fetching mention source: %s" % src_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 src_t = requests.get(src_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 src_html = BeautifulSoup(src_t.text, 'html.parser')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66 for link in src_html.find_all('a'):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67 href = link.get('href')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68 if href == tgt_url:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 break
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70 else:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 logger.error("Source '%s' doesn't link to target: %s" %
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 (src_url, tgt_url))
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 raise SourceDoesntLinkToTargetError()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 # Load the previous mentions and find any pre-existing mention from the
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 # source URL.
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 mention_path, mention_data = _load_page_mentions(req_page.page)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 for m in mention_data['mentions']:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 if m['source'] == src_url:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 logger.error("Duplicate mention found from: %s" % src_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81 raise DuplicateMentionError()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83 # Make the new mention.
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84 new_mention = {'source': src_url}
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
85
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
86 # Parse the microformats on the page, see if there's anything
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
87 # interesting we can use.
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
88 mf2_info = _get_mention_info_from_mf2(src_url, src_html)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
89 if mf2_info:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
90 new_mention.update(mf2_info)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
91
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
92 # Add the new mention.
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
93 mention_data['mentions'].append(new_mention)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
94
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
95 with open(mention_path, 'w', encoding='utf-8') as fp:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
96 json.dump(mention_data, fp)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
97 logger.info("Received webmention from: %s" % src_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
99
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
100 def _get_mention_info_from_mf2(base_url, bs_html):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
101 import mf2py
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
102 from urllib.parse import urljoin
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
103
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
104 mf2 = mf2py.parse(bs_html)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
105 mf2_items = mf2.get('items')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
106 if not mf2_items:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
107 return None
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
108
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
109 hentry = next(filter(
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
110 lambda i: 'h-entry' in i['type'],
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
111 mf2_items), None)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
112 if not hentry:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
113 return None
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
114
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
115 info = {}
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
116 hentry_props = hentry['properties']
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
117
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
118 pnames = hentry_props.get('name')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
119 if pnames:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
120 info['name'] = pnames[0]
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
121
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
122 urls = hentry_props.get('url')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
123 if urls:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
124 info['url'] = urljoin(base_url, urls[0])
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
125
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
126 pubdates = hentry_props.get('published')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
127 if pubdates:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
128 info['published'] = pubdates[0]
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
129
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
130 contents = hentry_props.get('content')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
131 if contents:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
132 info['content'] = contents[0]['html']
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
133
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
134 authors = hentry_props.get('author')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
135 if authors:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
136 hcard = next(filter(
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
137 lambda i: 'h-card' in i['type'],
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
138 authors), None)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
139 if hcard:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
140 hcard_props = hcard['properties']
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
141 hcard_names = hcard_props.get('name')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
142 if hcard_names:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
143 info['author_name'] = hcard_names[0]
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
144 hcard_photos = hcard_props.get('photo')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
145 if hcard_photos:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
146 info['author_photo'] = urljoin(base_url, hcard_photos[0])
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
147 hcard_urls = hcard_props.get('url')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
148 if hcard_urls:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
149 info['author_url'] = urljoin(base_url, hcard_urls[0])
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
150
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
151 return info
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
152
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
153
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
154 def _load_page_mentions(page):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
155 from piecrust.pathutil import ensure_dir
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
156
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
157 logger.debug("Loading page mentions for: %s" % page.content_spec)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
158 dirname, _ = os.path.splitext(page.content_spec)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
159 dirname += '-assets'
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
160 ensure_dir(dirname)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
161 mention_path = os.path.join(dirname, 'mentions.json')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
162 try:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
163 with open(mention_path, 'r', encoding='utf-8') as fp:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
164 return mention_path, json.load(fp)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
165 except IOError:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
166 return mention_path, {'mentions': []}