Mercurial > piecrust2

annotate piecrust/sources/blogarchives.py @ 1188:a7c43131d871

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(
author Ludovic Chabant <ludovic@chabant.com>
date Tue, 15 Jun 2021 22:36:23 -0700
parents 9f3e702a8a69
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
857
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
1 import time
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2 import logging
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 import datetime
857
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
4 import collections
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 from piecrust.data.filters import PaginationFilter, IFilterClause
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
6 from piecrust.dataproviders.pageiterator import (
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
7 PageIterator, HardCodedFilterIterator, DateSortIterator)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
8 from piecrust.page import Page
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
9 from piecrust.pipelines._pagebaker import PageBaker
1136
5f97b5b59dfe bake: Optimize cache handling for the baking process.
Ludovic Chabant <ludovic@chabant.com>
parents: 1047
diff changeset
10 from piecrust.pipelines._pagerecords import PagePipelineRecordEntry
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
11 from piecrust.pipelines.base import (
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
12 ContentPipeline,
1136
5f97b5b59dfe bake: Optimize cache handling for the baking process.
Ludovic Chabant <ludovic@chabant.com>
parents: 1047
diff changeset
13 create_job, get_record_name_for_source)
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
14 from piecrust.routing import RouteParameter
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
15 from piecrust.sources.base import ContentItem
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
16 from piecrust.sources.generator import GeneratorSourceBase
857
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
17 from piecrust.sources.list import ListSource
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
18
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
19
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
20 logger = logging.getLogger(__name__)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
21
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
22
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
23 _year_index = """---
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
24 layout: %(template)s
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
25 ---
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
26 """
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
27
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
28
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
29 class BlogArchivesSource(GeneratorSourceBase):
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
30 SOURCE_NAME = 'blog_archives'
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
31 DEFAULT_PIPELINE_NAME = 'blog_archives'
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
32
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
33 def __init__(self, app, name, config):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
34 super().__init__(app, name, config)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
35
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
36 tpl_name = config.get('template', '_year.html')
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
37 self._raw_item = _year_index % {'template': tpl_name}
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
38
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
39 def getSupportedRouteParameters(self):
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
40 return [RouteParameter('year', RouteParameter.TYPE_INT4)]
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
41
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
42 def findContentFromRoute(self, route_params):
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
43 year = route_params['year']
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
44 return ContentItem(
1144
9f3e702a8a69 bake: Give unique source specs to each taxonomy or blog archive page.
Ludovic Chabant <ludovic@chabant.com>
parents: 1136
diff changeset
45 '_index[%s]' % year,
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
46 {'route_params': {'year': year}})
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
47
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
48 def prepareRenderContext(self, ctx):
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
49 ctx.pagination_source = self.inner_source
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
50
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
51 route_params = ctx.page.source_metadata['route_params']
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
52 year = route_params.get('year')
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
53 if year is None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
54 raise Exception(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
55 "Can't find the archive year in the route metadata")
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
56 if type(year) is not int:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
57 raise Exception(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
58 "The route for generator '%s' should specify an integer "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
59 "parameter for 'year'." % self.name)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
60
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
61 flt = PaginationFilter()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
62 flt.addClause(IsFromYearFilterClause(year))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
63 ctx.pagination_filter = flt
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
64
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
65 ctx.custom_data['year'] = year
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
66
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
67 flt2 = PaginationFilter()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
68 flt2.addClause(IsFromYearFilterClause(year))
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
69 it = PageIterator(self.inner_source)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
70 it._simpleNonSortedWrap(HardCodedFilterIterator, flt2)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
71 it._wrapAsSort(DateSortIterator, reverse=False)
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
72 ctx.custom_data['archives'] = it
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
73
857
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
74 ctx.custom_data['monthly_archives'] = _MonthlyArchiveData(
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
75 self.inner_source, year)
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
76
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
77
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
78 class IsFromYearFilterClause(IFilterClause):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
79 def __init__(self, year):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
80 self.year = year
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
81
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
82 def pageMatches(self, fil, page):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
83 return (page.datetime.year == self.year)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
84
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
85
857
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
86 class _MonthlyArchiveData(collections.abc.Mapping):
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
87 def __init__(self, inner_source, year):
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
88 self._inner_source = inner_source
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
89 self._year = year
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
90 self._months = None
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
91
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
92 def __iter__(self):
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
93 self._load()
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
94 return iter(self._months)
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
95
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
96 def __len__(self):
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
97 self._load()
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
98 return len(self._months)
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
99
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
100 def __getitem__(self, i):
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
101 self._load()
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
102 return self._months[i]
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
103
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
104 def _load(self):
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
105 if self._months is not None:
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
106 return
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
107
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
108 month_index = {}
905
1d0364614665 internal: Sources can cache their pages in addition to their items.
Ludovic Chabant <ludovic@chabant.com>
parents: 877
diff changeset
109 for page in self._inner_source.getAllPages():
857
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
110 if page.datetime.year != self._year:
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
111 continue
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
112
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
113 month = page.datetime.month
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
114
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
115 posts_this_month = month_index.get(month)
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
116 if posts_this_month is None:
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
117 posts_this_month = []
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
118 month_index[month] = posts_this_month
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
119 posts_this_month.append(page.content_item)
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
120
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
121 self._months = []
1047
ba809c221a27 data: Properly order month names in blog archives.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
122 for m in sorted(month_index.keys()):
857
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
123 timestamp = time.mktime((self._year, m, 1, 0, 0, 0, 0, 0, -1))
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
124
1047
ba809c221a27 data: Properly order month names in blog archives.
Ludovic Chabant <ludovic@chabant.com>
parents: 989
diff changeset
125 ptm = month_index[m]
857
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
126 it = PageIterator(ListSource(self._inner_source, ptm))
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
127 it._wrapAsSort(DateSortIterator, reverse=False)
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
128
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
129 self._months.append({
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
130 'timestamp': timestamp,
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
131 'posts': it
d231a10d18f9 refactor: Make the data providers and blog archives source functional.
Ludovic Chabant <ludovic@chabant.com>
parents: 856
diff changeset
132 })
852
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
133
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
134
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
135 class BlogArchivesPipelineRecordEntry(PagePipelineRecordEntry):
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
136 def __init__(self):
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
137 super().__init__()
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
138 self.year = None
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
139
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
140
854
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
141 class BlogArchivesPipeline(ContentPipeline):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources.
Ludovic Chabant <ludovic@chabant.com>
parents: 853
diff changeset
142 PIPELINE_NAME = 'blog_archives'
877
d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps.
Ludovic Chabant <ludovic@chabant.com>
parents: 876
diff changeset
143 PASS_NUM = 10
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
144 RECORD_ENTRY_CLASS = BlogArchivesPipelineRecordEntry
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
145
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
146 def __init__(self, source, ctx):
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
147 if not isinstance(source, BlogArchivesSource):
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
148 raise Exception("The blog archives pipeline only supports blog "
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
149 "archives content sources.")
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
150
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
151 super().__init__(source, ctx)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
152 self.inner_source = source.inner_source
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
153 self._tpl_name = source.config['template']
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
154 self._all_years = None
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
155 self._dirty_years = None
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
156 self._pagebaker = None
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
157
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
158 def initialize(self):
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
159 self._pagebaker = PageBaker(self.app,
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
160 self.ctx.out_dir,
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
161 force=self.ctx.force)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
162 self._pagebaker.startWriterQueue()
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
163
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
164 def shutdown(self):
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
165 self._pagebaker.stopWriterQueue()
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
166
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
167 def createJobs(self, ctx):
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
168 logger.debug("Building blog archives for: %s" %
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
169 self.inner_source.name)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
170 self._buildDirtyYears(ctx)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
171 logger.debug("Got %d dirty years out of %d." %
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
172 (len(self._dirty_years), len(self._all_years)))
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
173
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
174 jobs = []
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
175 rec_fac = self.createRecordEntry
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
176 current_record = ctx.current_record
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
177
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
178 for y in self._dirty_years:
1144
9f3e702a8a69 bake: Give unique source specs to each taxonomy or blog archive page.
Ludovic Chabant <ludovic@chabant.com>
parents: 1136
diff changeset
179 item_spec = '_index[%04d]' % y
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
180
1144
9f3e702a8a69 bake: Give unique source specs to each taxonomy or blog archive page.
Ludovic Chabant <ludovic@chabant.com>
parents: 1136
diff changeset
181 jobs.append(create_job(self, item_spec,
9f3e702a8a69 bake: Give unique source specs to each taxonomy or blog archive page.
Ludovic Chabant <ludovic@chabant.com>
parents: 1136
diff changeset
182 year=y))
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
183
1144
9f3e702a8a69 bake: Give unique source specs to each taxonomy or blog archive page.
Ludovic Chabant <ludovic@chabant.com>
parents: 1136
diff changeset
184 entry = rec_fac(item_spec)
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
185 current_record.addEntry(entry)
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
186
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
187 if len(jobs) > 0:
1136
5f97b5b59dfe bake: Optimize cache handling for the baking process.
Ludovic Chabant <ludovic@chabant.com>
parents: 1047
diff changeset
188 return jobs, "archive"
5f97b5b59dfe bake: Optimize cache handling for the baking process.
Ludovic Chabant <ludovic@chabant.com>
parents: 1047
diff changeset
189 return None, None
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
190
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
191 def run(self, job, ctx, result):
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
192 year = job['year']
1144
9f3e702a8a69 bake: Give unique source specs to each taxonomy or blog archive page.
Ludovic Chabant <ludovic@chabant.com>
parents: 1136
diff changeset
193 content_item = ContentItem('_index[%04d]' % year,
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
194 {'year': year,
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
195 'route_params': {'year': year}})
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
196 page = Page(self.source, content_item)
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
197
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
198 prev_entry = ctx.previous_entry
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
199 rdr_subs = self._pagebaker.bake(page, prev_entry)
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
200
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
201 result['subs'] = rdr_subs
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
202 result['year'] = page.source_metadata['year']
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
203
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
204 def handleJobResult(self, result, ctx):
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
205 existing = ctx.record_entry
1136
5f97b5b59dfe bake: Optimize cache handling for the baking process.
Ludovic Chabant <ludovic@chabant.com>
parents: 1047
diff changeset
206 existing.subs = result['subs']
989
8adc27285d93 bake: Big pass on bake performance.
Ludovic Chabant <ludovic@chabant.com>
parents: 905
diff changeset
207 existing.year = result['year']
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
208
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
209 def postJobRun(self, ctx):
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
210 # Create bake entries for the years that were *not* dirty.
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
211 # Otherwise, when checking for deleted pages, we would not find any
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
212 # outputs and would delete those files.
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
213 all_str_years = [str(y) for y in self._all_years]
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
214 for prev, cur in ctx.record_history.diffs:
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
215 if prev and not cur:
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
216 y = prev.year
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
217 if y in all_str_years:
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
218 logger.debug(
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
219 "Creating unbaked entry for year %s archive." % y)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
220 cur.year = y
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
221 cur.out_paths = list(prev.out_paths)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
222 cur.errors = list(prev.errors)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
223 else:
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
224 logger.debug(
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
225 "No page references year %s anymore." % y)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
226
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
227 def _buildDirtyYears(self, ctx):
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
228 all_years = set()
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
229 dirty_years = set()
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
230
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
231 record_name = get_record_name_for_source(self.inner_source)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
232 current_records = ctx.record_histories.current
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
233 cur_rec = current_records.getRecord(record_name)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
234 for cur_entry in cur_rec.getEntries():
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
235 dt = datetime.datetime.fromtimestamp(cur_entry.timestamp)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
236 all_years.add(dt.year)
1136
5f97b5b59dfe bake: Optimize cache handling for the baking process.
Ludovic Chabant <ludovic@chabant.com>
parents: 1047
diff changeset
237 if cur_entry.hasFlag(
5f97b5b59dfe bake: Optimize cache handling for the baking process.
Ludovic Chabant <ludovic@chabant.com>
parents: 1047
diff changeset
238 PagePipelineRecordEntry.FLAG_SEGMENTS_RENDERED):
856
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
239 dirty_years.add(dt.year)
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
240
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
241 self._all_years = all_years
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
242 self._dirty_years = dirty_years
9bb22bbe093c refactor: Make the blog archives functional again.
Ludovic Chabant <ludovic@chabant.com>
parents: 854
diff changeset
243