piecrust2: piecrust/pipelines/base.py annotate

annotate piecrust/pipelines/base.py @ 1051:971b4d67e82a

serve: Fix problems with assets disappearing between servings. When an asset file changes, its source's pipeline is re-run. But that created a bake record that only had that pipeline's output, so the other outputs were incorrectly considered empty and therefore any stray files were removed. Now we copy over bake records for the pipelines we don't run.

author	Ludovic Chabant <ludovic@chabant.com>
date	Fri, 26 Jan 2018 18:05:02 -0800
parents	fa489c5e829e
children	5f97b5b59dfe

rev	line source
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	1 import os.path
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import logging
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	3 from werkzeug.utils import cached_property
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	4 from piecrust.configuration import ConfigurationError
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	5 from piecrust.sources.base import ContentItem
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	7
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8 logger = logging.getLogger(__name__)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	9
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	10
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	11 class PipelineContext:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12 """ The context for running a content pipeline.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13 """
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	14 def __init__(self, out_dir, *,
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	15 worker_id=-1, force=None):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	16 self.out_dir = out_dir
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	17 self.worker_id = worker_id
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	18 self.force = force
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	19
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20 @property
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21 def is_worker(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	22 """ Returns `True` if the content pipeline is running inside
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	23 a worker process, and this is the first one.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24 """
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	25 return self.worker_id >= 0
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27 @property
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 def is_main_process(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29 """ Returns `True` is the content pipeline is running inside
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	30 the main process (and not a worker process). This is the case
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31 if there are no worker processes at all.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32 """
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	33 return self.worker_id < 0
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	34
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	35
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	36 class _PipelineMasterProcessJobContextBase:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	37 def __init__(self, record_name, record_histories):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	38 self.record_name = record_name
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	39 self.record_histories = record_histories
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	40
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	41 @property
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	42 def previous_record(self):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	43 return self.record_histories.getPreviousRecord(self.record_name)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	44
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	45 @property
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	46 def current_record(self):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	47 return self.record_histories.getCurrentRecord(self.record_name)
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	48
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	49
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	50 class PipelineJobCreateContext(_PipelineMasterProcessJobContextBase):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	51 """ Context for creating pipeline baking jobs.
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	52
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	53 This is run on the master process, so it can access both the
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	54 previous and current records.
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	55 """
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	56 def __init__(self, pass_num, record_name, record_histories):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	57 super().__init__(record_name, record_histories)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	58 self.pass_num = pass_num
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	59
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	60
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	61 class PipelineJobValidateContext(_PipelineMasterProcessJobContextBase):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	62 """ Context for validating jobs on subsequent step runs (i.e. validating
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	63 the list of jobs to run starting with the second step).
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	64
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	65 This is run on the master process, so it can access both the
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	66 previous and current records.
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	67 """
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	68 def __init__(self, pass_num, step_num, record_name, record_histories):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	69 super().__init__(record_name, record_histories)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	70 self.pass_num = pass_num
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	71 self.step_num = step_num
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	72
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	73
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	74 class PipelineJobRunContext:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	75 """ Context for running pipeline baking jobs.
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	76
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	77 This is run on the worker processes, so it can only access the
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	78 previous records.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	79 """
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	80 def __init__(self, job, record_name, previous_records):
876 d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	81 self.job = job
d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	82 self.record_name = record_name
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	83 self.previous_records = previous_records
876 d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	84
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	85 @cached_property
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	86 def record_entry_spec(self):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	87 return self.job.get('record_entry_spec',
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	88 self.job['job_spec'][1])
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	89
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	90 @cached_property
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	91 def previous_record(self):
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	92 return self.previous_records.getRecord(self.record_name)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	93
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	94 @cached_property
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	95 def previous_entry(self):
876 d1095774bfcf refactor: Fix some issues with record/cache entry collisions, add counters. Ludovic Chabant <ludovic@chabant.com> parents: 855 diff changeset	96 return self.previous_record.getEntry(self.record_entry_spec)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	97
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	98
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	99 class PipelineJobResultHandleContext:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	100 """ The context for handling the result from a job that ran in a
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	101 worker process.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	102
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	103 This is run on the master process, so it can access the current
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	104 record.
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	105 """
1015 fa489c5e829e bake: Load pages in parallel again. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	106 def __init__(self, record, job, pass_num, step_num):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	107 self.record = record
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	108 self.job = job
1015 fa489c5e829e bake: Load pages in parallel again. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	109 self.pass_num = pass_num
877 d6d35b2efd04 bake: Rename "pass" to "step" and make the page pipeline use different steps. Ludovic Chabant <ludovic@chabant.com> parents: 876 diff changeset	110 self.step_num = step_num
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	111
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	112 @cached_property
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	113 def record_entry(self):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	114 record_entry_spec = self.job.get('record_entry_spec',
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	115 self.job['job_spec'][1])
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	116 return self.record.getEntry(record_entry_spec)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	117
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	118
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	119 class PipelinePostJobRunContext:
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	120 def __init__(self, record_history):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	121 self.record_history = record_history
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	122
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	123
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	124 class PipelineDeletionContext:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	125 def __init__(self, record_history):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	126 self.record_history = record_history
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	127
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	128
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	129 class PipelineCollapseRecordContext:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	130 def __init__(self, record_history):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	131 self.record_history = record_history
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	132
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	133
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	134 class ContentPipeline:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	135 """ A pipeline that processes content from a `ContentSource`.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	136 """
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	137 PIPELINE_NAME = None
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	138 RECORD_ENTRY_CLASS = None
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	139 PASS_NUM = 0
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	140
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	141 def __init__(self, source, ctx):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	142 self.source = source
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	143 self.ctx = ctx
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	144 self.record_name = '%s@%s' % (source.name, self.PIPELINE_NAME)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	145
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	146 app = source.app
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	147 tmp_dir = app.cache_dir
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	148 if not tmp_dir:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	149 import tempfile
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150 tmp_dir = os.path.join(tempfile.gettempdir(), 'piecrust')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	151 self.tmp_dir = os.path.join(tmp_dir, self.PIPELINE_NAME)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	152
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	153 @property
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	154 def app(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	155 return self.source.app
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	156
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	157 def initialize(self):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	158 pass
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	159
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	160 def createJobs(self, ctx):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	161 return [
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	162 create_job(self, item.spec)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	163 for item in self.source.getAllContents()]
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	164
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	165 def createRecordEntry(self, item_spec):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	166 entry_class = self.RECORD_ENTRY_CLASS
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	167 record_entry = entry_class()
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	168 record_entry.item_spec = item_spec
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	169 return record_entry
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	170
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	171 def handleJobResult(self, result, ctx):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	172 raise NotImplementedError()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	173
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	174 def validateNextStepJobs(self, jobs, ctx):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	175 pass
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	176
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	177 def run(self, job, ctx, result):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	178 raise NotImplementedError()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	179
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	180 def postJobRun(self, ctx):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	181 pass
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	182
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	183 def getDeletions(self, ctx):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	184 pass
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	185
853 f070a4fc033c core: Continue PieCrust3 refactor, simplify pages. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	186 def collapseRecords(self, ctx):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	187 pass
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	188
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	189 def shutdown(self):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	190 pass
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	191
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	192
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	193 def create_job(pipeline, item_spec, **kwargs):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	194 job = {
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	195 'job_spec': (pipeline.source.name, item_spec)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	196 }
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	197 job.update(kwargs)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	198 return job
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	199
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	200
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	201 def content_item_from_job(pipeline, job):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	202 return pipeline.source.findContentFromSpec(job['job_spec'][1])
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	203
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	204
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	205 def get_record_name_for_source(source):
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	206 ppname = get_pipeline_name_for_source(source)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	207 return '%s@%s' % (source.name, ppname)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	208
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	209
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	210 def get_pipeline_name_for_source(source):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	211 pname = source.config['pipeline']
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	212 if not pname:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	213 pname = source.DEFAULT_PIPELINE_NAME
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	214 if not pname:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	215 raise ConfigurationError(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	216 "Source '%s' doesn't specify any pipeline." % source.name)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	217 return pname
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	218
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	219
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	220 class PipelineManager:
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	221 def __init__(self, app, out_dir, *,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	222 record_histories=None, worker_id=-1, force=False):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	223 self.app = app
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	224 self.record_histories = record_histories
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	225 self.out_dir = out_dir
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	226 self.worker_id = worker_id
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	227 self.force = force
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	228
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	229 self._pipeline_classes = {}
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	230 for pclass in app.plugin_loader.getPipelines():
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	231 self._pipeline_classes[pclass.PIPELINE_NAME] = pclass
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	232
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	233 self._pipelines = {}
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	234
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	235 def getPipeline(self, source_name):
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	236 return self.getPipelineInfo(source_name).pipeline
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	237
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	238 def getPipelineInfo(self, source_name):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	239 return self._pipelines[source_name]
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	240
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	241 def getPipelineInfos(self):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	242 return self._pipelines.values()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	243
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	244 def createPipeline(self, source):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	245 if source.name in self._pipelines:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	246 raise ValueError("Pipeline for source '%s' was already created." %
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	247 source.name)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	248
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	249 pname = get_pipeline_name_for_source(source)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	250 ppctx = PipelineContext(self.out_dir,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	251 worker_id=self.worker_id, force=self.force)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	252 pp = self._pipeline_classes[pname](source, ppctx)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	253 pp.initialize()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	254
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	255 record_history = None
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	256 if self.record_histories:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	257 record_history = self.record_histories.getHistory(pp.record_name)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	258
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	259 info = _PipelineInfo(pp, record_history)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	260 self._pipelines[source.name] = info
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	261 return info
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	262
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	263 def postJobRun(self):
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	264 for ppinfo in self.getPipelineInfos():
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	265 ppinfo.record_history.build()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	266
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	267 for ppinfo in self.getPipelineInfos():
855 448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	268 ctx = PipelinePostJobRunContext(ppinfo.record_history)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	269 ppinfo.pipeline.postJobRun(ctx)
448710d84121 refactor: Get the taxonomy support back to a functional state. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	270
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	271 def deleteStaleOutputs(self):
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	272 for ppinfo in self.getPipelineInfos():
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	273 ctx = PipelineDeletionContext(ppinfo.record_history)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	274 to_delete = ppinfo.pipeline.getDeletions(ctx)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	275 current_record = ppinfo.record_history.current
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	276 if to_delete is not None:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	277 for path, reason in to_delete:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	278 logger.debug("Removing '%s': %s" % (path, reason))
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	279 current_record.deleted_out_paths.append(path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	280 try:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	281 os.remove(path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	282 except FileNotFoundError:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	283 pass
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	284 logger.info('[delete] %s' % path)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	285
1051 971b4d67e82a serve: Fix problems with assets disappearing between servings. Ludovic Chabant <ludovic@chabant.com> parents: 1015 diff changeset	286 def collapseRecords(self, keep_unused_records=False):
971b4d67e82a serve: Fix problems with assets disappearing between servings. Ludovic Chabant <ludovic@chabant.com> parents: 1015 diff changeset	287 seen_records = []
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	288 for ppinfo in self.getPipelineInfos():
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	289 ctx = PipelineCollapseRecordContext(ppinfo.record_history)
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	290 ppinfo.pipeline.collapseRecords(ctx)
1051 971b4d67e82a serve: Fix problems with assets disappearing between servings. Ludovic Chabant <ludovic@chabant.com> parents: 1015 diff changeset	291 seen_records.append(ppinfo.pipeline.record_name)
971b4d67e82a serve: Fix problems with assets disappearing between servings. Ludovic Chabant <ludovic@chabant.com> parents: 1015 diff changeset	292
971b4d67e82a serve: Fix problems with assets disappearing between servings. Ludovic Chabant <ludovic@chabant.com> parents: 1015 diff changeset	293 if keep_unused_records:
971b4d67e82a serve: Fix problems with assets disappearing between servings. Ludovic Chabant <ludovic@chabant.com> parents: 1015 diff changeset	294 cur_recs = self.record_histories.current
971b4d67e82a serve: Fix problems with assets disappearing between servings. Ludovic Chabant <ludovic@chabant.com> parents: 1015 diff changeset	295 prev_recs = self.record_histories.previous
971b4d67e82a serve: Fix problems with assets disappearing between servings. Ludovic Chabant <ludovic@chabant.com> parents: 1015 diff changeset	296 for prev_rec in prev_recs.records:
971b4d67e82a serve: Fix problems with assets disappearing between servings. Ludovic Chabant <ludovic@chabant.com> parents: 1015 diff changeset	297 if prev_rec.name in seen_records:
971b4d67e82a serve: Fix problems with assets disappearing between servings. Ludovic Chabant <ludovic@chabant.com> parents: 1015 diff changeset	298 continue
971b4d67e82a serve: Fix problems with assets disappearing between servings. Ludovic Chabant <ludovic@chabant.com> parents: 1015 diff changeset	299
971b4d67e82a serve: Fix problems with assets disappearing between servings. Ludovic Chabant <ludovic@chabant.com> parents: 1015 diff changeset	300 logger.debug("Keeping record: %s" % prev_rec.name)
971b4d67e82a serve: Fix problems with assets disappearing between servings. Ludovic Chabant <ludovic@chabant.com> parents: 1015 diff changeset	301 cur_recs.records.append(prev_rec)
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	302
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	303 def shutdownPipelines(self):
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	304 for ppinfo in self.getPipelineInfos():
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	305 ppinfo.pipeline.shutdown()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	306
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	307 self._pipelines = {}
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	308
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	309
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	310 class _PipelineInfo:
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	311 def __init__(self, pipeline, record_history):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	312 self.pipeline = pipeline
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	313 self.record_history = record_history
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	314 self.userdata = None
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	315
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	316 @property
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	317 def source(self):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	318 return self.pipeline.source
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	319
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	320 @property
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	321 def current_record(self):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	322 if self.record_history is not None:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	323 return self.record_history.current
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	324 raise Exception("The current record is not available.")
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	325
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 877 diff changeset	326 @property
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	327 def pipeline_name(self):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	328 return self.pipeline.PIPELINE_NAME
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 853 diff changeset	329

Mercurial > piecrust2

annotate piecrust/pipelines/base.py @ 1051:971b4d67e82a