Mercurial > piecrust2

annotate piecrust/tasks/mentions.py @ 1188:a7c43131d871

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 15 Jun 2021 22:36:23 -0700
parents 8af2ea1f5c34
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1114
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 import os
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import os.path
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import json
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 import logging
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 from piecrust.tasks.base import TaskRunner
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
7
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
8 logger = logging.getLogger(__name__)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
9
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
10
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
11 class InvalidMentionTargetError(Exception):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
12 pass
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
13
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
15 class SourceDoesntLinkToTargetError(Exception):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
16 pass
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
17
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19 class DuplicateMentionError(Exception):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 pass
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
23 class MentionTaskRunner(TaskRunner):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
24 TASK_TYPE = 'mention'
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
25
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
26 def runTask(self, data, ctx):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
27 import json
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
28 import requests
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
29 from bs4 import BeautifulSoup
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 from piecrust.app import PieCrustFactory
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
31 from piecrust.serving.util import get_requested_page
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 src_url = data['source']
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 tgt_url = data['target']
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
36 # Find if we have a page at the target URL. To do that we need to spin
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
37 # up a PieCrust app that knows how the website works. Because the
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
38 # website might have been baked with custom settings (usually the site
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
39 # root URL) there's a good chance we need to apply some variants, which
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
40 # the user can specify in the config.
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
41 pcappfac = PieCrustFactory(self.app.root_dir,
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
42 cache_key='webmention')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
43 wmcfg = self.app.config.get('webmention')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
44 if wmcfg.get('config_variant'):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
45 pcappfac.config_variants = [wmcfg.get('config_variant')]
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
46 if wmcfg.get('config_variants'):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47 pcappfac.config_variants = list(wmcfg.get('config_variants'))
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 if wmcfg.get('config_values'):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
49 pcappfac.config_values = list(wmcfg.get('config_values').items())
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50 pcapp = pcappfac.create()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
51 logger.debug("Locating page: %s" % tgt_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
52 try:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 req_page = get_requested_page(pcapp, tgt_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 if req_page.page is None:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 raise InvalidMentionTargetError()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56 except Exception as ex:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 logger.error("Can't check webmention target page: %s" % tgt_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 logger.exception(ex)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 raise InvalidMentionTargetError()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61 # Grab the source URL's contents and see if anything references the
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62 # target (ours) URL.
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63 logger.debug("Fetching mention source: %s" % src_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64 src_t = requests.get(src_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 src_html = BeautifulSoup(src_t.text, 'html.parser')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66 for link in src_html.find_all('a'):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67 href = link.get('href')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68 if href == tgt_url:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
69 break
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
70 else:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
71 logger.error("Source '%s' doesn't link to target: %s" %
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 (src_url, tgt_url))
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73 raise SourceDoesntLinkToTargetError()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
74
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
75 # Load the previous mentions and find any pre-existing mention from the
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76 # source URL.
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77 mention_path, mention_data = _load_page_mentions(req_page.page)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 for m in mention_data['mentions']:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 if m['source'] == src_url:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 logger.error("Duplicate mention found from: %s" % src_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81 raise DuplicateMentionError()
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83 # Make the new mention.
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84 new_mention = {'source': src_url}
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
85
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
86 # Parse the microformats on the page, see if there's anything
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
87 # interesting we can use.
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
88 mf2_info = _get_mention_info_from_mf2(src_url, src_html)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
89 if mf2_info:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
90 new_mention.update(mf2_info)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
91
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
92 # Add the new mention.
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
93 mention_data['mentions'].append(new_mention)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
94
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
95 with open(mention_path, 'w', encoding='utf-8') as fp:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
96 json.dump(mention_data, fp)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
97 logger.info("Received webmention from: %s" % src_url)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
98
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
99
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
100 def _get_mention_info_from_mf2(base_url, bs_html):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
101 import mf2py
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
102 from urllib.parse import urljoin
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
103
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
104 mf2 = mf2py.parse(bs_html)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
105 mf2_items = mf2.get('items')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
106 if not mf2_items:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
107 return None
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
108
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
109 hentry = next(filter(
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
110 lambda i: 'h-entry' in i['type'],
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
111 mf2_items), None)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
112 if not hentry:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
113 return None
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
114
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
115 info = {}
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
116 hentry_props = hentry['properties']
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
117
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
118 pnames = hentry_props.get('name')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
119 if pnames:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
120 info['name'] = pnames[0]
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
121
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
122 urls = hentry_props.get('url')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
123 if urls:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
124 info['url'] = urljoin(base_url, urls[0])
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
125
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
126 pubdates = hentry_props.get('published')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
127 if pubdates:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
128 info['published'] = pubdates[0]
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
129
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
130 contents = hentry_props.get('content')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
131 if contents:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
132 info['content'] = contents[0]['html']
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
133
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
134 authors = hentry_props.get('author')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
135 if authors:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
136 hcard = next(filter(
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
137 lambda i: 'h-card' in i['type'],
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
138 authors), None)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
139 if hcard:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
140 hcard_props = hcard['properties']
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
141 hcard_names = hcard_props.get('name')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
142 if hcard_names:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
143 info['author_name'] = hcard_names[0]
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
144 hcard_photos = hcard_props.get('photo')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
145 if hcard_photos:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
146 info['author_photo'] = urljoin(base_url, hcard_photos[0])
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
147 hcard_urls = hcard_props.get('url')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
148 if hcard_urls:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
149 info['author_url'] = urljoin(base_url, hcard_urls[0])
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
150
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
151 return info
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
152
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
153
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
154 def _load_page_mentions(page):
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
155 from piecrust.pathutil import ensure_dir
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
156
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
157 logger.debug("Loading page mentions for: %s" % page.content_spec)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
158 dirname, _ = os.path.splitext(page.content_spec)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
159 dirname += '-assets'
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
160 ensure_dir(dirname)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
161 mention_path = os.path.join(dirname, 'mentions.json')
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
162 try:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
163 with open(mention_path, 'r', encoding='utf-8') as fp:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
164 return mention_path, json.load(fp)
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
165 except IOError:
8af2ea1f5c34 tasks: Add new `tasks` command and infrastructure, with `mention` task.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
166 return mention_path, {'mentions': []}