piecrust2: piecrust/workerpool.py annotate

annotate piecrust/workerpool.py @ 1188:a7c43131d871

bake: Fix file write flushing problem with Python 3.8+ Writing the cache files fails in Python 3.8 because it looks like flushing behaviour has changed. We need to explicitly flush. And even then, in very rare occurrences, it looks like it can still run into racing conditions, so we do a very hacky and ugly "retry" loop when fetching cached data :(

author	Ludovic Chabant <ludovic@chabant.com>
date	Tue, 15 Jun 2021 22:36:23 -0700
parents	aa4f1e04cf3e
children

rev	line source
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	1 import io
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import sys
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	4 import time
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import logging
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 import threading
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	7 import traceback
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8 import multiprocessing
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	9 from piecrust.environment import ExecutionStats
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	10
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	11
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12 logger = logging.getLogger(__name__)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	14 use_fastqueue = False
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	15 use_fastpickle = False
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	16 use_msgpack = False
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	17 use_marshall = False
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	18 use_json = False
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	19
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21 class IWorker(object):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	22 """ Interface for a pool worker.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	23 """
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24 def initialize(self):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	25 raise NotImplementedError()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27 def process(self, job):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 raise NotImplementedError()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	30 def getStats(self):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31 return None
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32
702 c62d83e17abf bake: Some more optimizations. Ludovic Chabant <ludovic@chabant.com> parents: 697 diff changeset	33 def shutdown(self):
c62d83e17abf bake: Some more optimizations. Ludovic Chabant <ludovic@chabant.com> parents: 697 diff changeset	34 pass
c62d83e17abf bake: Some more optimizations. Ludovic Chabant <ludovic@chabant.com> parents: 697 diff changeset	35
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	37 def _get_worker_exception_data(wid):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	38 t, v, tb = sys.exc_info()
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	39 return {
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	40 'wid': wid,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	41 'type': str(t),
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	42 'value': '\n'.join(_get_errors(v)),
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	43 'traceback': ''.join(traceback.format_exception(t, v, tb))
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	44 }
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	45
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	46
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	47 def _get_errors(ex):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	48 errors = []
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	49 while ex is not None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	50 msg = str(ex)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	51 errors.append(msg)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	52 ex = ex.__cause__
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	53 return errors
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	54
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	55
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	56 TASK_JOB = 0
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	57 TASK_JOB_BATCH = 1
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	58 TASK_END = 2
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	59 _TASK_ABORT_WORKER = 10
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	60 _CRITICAL_WORKER_ERROR = 11
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	61
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63 def worker_func(params):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64 if params.is_profiling:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	65 try:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	66 import cProfile as profile
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	67 except ImportError:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 import profile
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	70 params.is_profiling = False
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71 name = params.worker_class.__name__
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	72 profile.runctx('_real_worker_func(params)',
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	73 globals(), locals(),
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	74 filename='%s-%d.prof' % (name, params.wid))
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75 else:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76 _real_worker_func(params)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	77
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	78
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	79 def _real_worker_func(params):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	80 try:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	81 _real_worker_func_unsafe(params)
1046 aa4f1e04cf3e serve: Don't spam the output when hitting CTRL+C while processing assets. Ludovic Chabant <ludovic@chabant.com> parents: 1019 diff changeset	82 except (KeyboardInterrupt, SystemExit):
aa4f1e04cf3e serve: Don't spam the output when hitting CTRL+C while processing assets. Ludovic Chabant <ludovic@chabant.com> parents: 1019 diff changeset	83 # Return silently
aa4f1e04cf3e serve: Don't spam the output when hitting CTRL+C while processing assets. Ludovic Chabant <ludovic@chabant.com> parents: 1019 diff changeset	84 pass
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	85 except Exception as ex:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	86 logger.exception(ex)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	87 msg = ("CRITICAL ERROR IN WORKER %d\n%s" % (params.wid, str(ex)))
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	88 params.outqueue.put((
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	89 _CRITICAL_WORKER_ERROR, None, False, params.wid, msg))
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	90
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	91
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	92 def _pre_parse_pytest_args():
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	93 # If we are unit-testing, we need to translate our test logging
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	94 # arguments into something Chef can understand.
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	95 import argparse
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	96 parser = argparse.ArgumentParser()
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	97 # This is adapted from our `conftest.py`.
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	98 parser.add_argument('--log-debug', action='store_true')
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	99 parser.add_argument('--log-file')
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	100 res, _ = parser.parse_known_args(sys.argv[1:])
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	101
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	102 chef_args = []
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	103 if res.log_debug:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	104 chef_args.append('--debug')
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	105 if res.log_file:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	106 chef_args += ['--log', res.log_file]
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	107
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	108 root_logger = logging.getLogger()
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	109 while len(root_logger.handlers) > 0:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	110 root_logger.removeHandler(root_logger.handlers[0])
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	111
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	112 from piecrust.main import _pre_parse_chef_args
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	113 _pre_parse_chef_args(chef_args)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	114
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	115
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	116 def _real_worker_func_unsafe(params):
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	117 init_start_time = time.perf_counter()
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	118
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	119 wid = params.wid
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	120
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	121 stats = ExecutionStats()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	122 stats.registerTimer('WorkerInit')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	123
500 22a230d99621 bake: Fix logging configuration for multi-processing on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	124 # In a context where `multiprocessing` is using the `spawn` forking model,
22a230d99621 bake: Fix logging configuration for multi-processing on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	125 # the new process doesn't inherit anything, so we lost all our logging
22a230d99621 bake: Fix logging configuration for multi-processing on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	126 # configuration here. Let's set it up again.
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	127 if (hasattr(multiprocessing, 'get_start_method') and
566 8073ae8cb164 bake: Don't re-setup logging for workers unless we're sure we need it. Ludovic Chabant <ludovic@chabant.com> parents: 500 diff changeset	128 multiprocessing.get_start_method() == 'spawn'):
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	129 if not params.is_unit_testing:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	130 from piecrust.main import _pre_parse_chef_args
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	131 _pre_parse_chef_args(sys.argv[1:])
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	132 else:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	133 _pre_parse_pytest_args()
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	134 elif params.is_unit_testing:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	135 _pre_parse_pytest_args()
500 22a230d99621 bake: Fix logging configuration for multi-processing on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	136
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	137 from piecrust.main import ColoredFormatter
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	138 root_logger = logging.getLogger()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	139 root_logger.handlers[0].setFormatter(ColoredFormatter(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	140 ('[W-%d]' % wid) + '[%(name)s] %(message)s'))
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	141
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	142 logger.debug("Worker %d initializing..." % wid)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	143
693 d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	144 # We don't need those.
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	145 params.inqueue._writer.close()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	146 params.outqueue._reader.close()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	147
693 d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	148 # Initialize the underlying worker class.
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	149 w = params.worker_class(*params.initargs)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150 w.wid = wid
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	151 try:
8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	152 w.initialize()
8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	153 except Exception as ex:
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	154 logger.error("Worker %d failed to initialize." % wid)
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	155 logger.exception(ex)
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	156 raise
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	157
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	158 stats.stepTimerSince('WorkerInit', init_start_time)
693 d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	159
d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	160 # Start pumping!
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	161 completed = 0
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	162 time_in_get = 0
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	163 time_in_put = 0
1019 bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	164 is_first_get = True
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	165 get = params.inqueue.get
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	166 put = params.outqueue.put
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	167
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	168 while True:
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	169 get_start_time = time.perf_counter()
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	170 task = get()
1019 bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	171 if not is_first_get:
bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	172 time_in_get += (time.perf_counter() - get_start_time)
bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	173 else:
bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	174 is_first_get = False
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	175
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	176 task_type, task_data = task
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	177
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	178 # Job task(s)... just do it.
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	179 if task_type == TASK_JOB or task_type == TASK_JOB_BATCH:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	180
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	181 task_data_list = task_data
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	182 if task_type == TASK_JOB:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	183 task_data_list = [task_data]
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	184
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	185 result_list = []
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	186
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	187 for td in task_data_list:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	188 try:
1008 09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	189 worker_res = w.process(td)
09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	190 result_list.append((td, worker_res, True))
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	191 except Exception as e:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	192 logger.debug(
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	193 "Error processing job, sending exception to main process:")
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	194 logger.debug(traceback.format_exc())
1019 bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	195 error_res = _get_worker_exception_data(wid)
1008 09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	196 result_list.append((td, error_res, False))
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	197
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	198 res = (task_type, wid, result_list)
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	199 put_start_time = time.perf_counter()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	200 put(res)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	201 time_in_put += (time.perf_counter() - put_start_time)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	202
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	203 completed += len(task_data_list)
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	204
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	205 # End task... gather stats to send back to the main process.
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	206 elif task_type == TASK_END:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	207 logger.debug("Worker %d got end task, exiting." % wid)
1019 bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	208 stats.registerTimer('Worker_%d_TaskGet' % wid, time=time_in_get)
bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	209 stats.registerTimer('Worker_all_TaskGet', time=time_in_get)
bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	210 stats.registerTimer('Worker_%d_ResultPut' % wid, time=time_in_put)
bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	211 stats.registerTimer('Worker_all_ResultPut', time=time_in_put)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	212 try:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	213 stats.mergeStats(w.getStats())
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	214 stats_data = stats.toData()
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	215 rep = (task_type, wid, [(task_data, (wid, stats_data), True)])
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	216 except Exception as e:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	217 logger.debug(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	218 "Error getting report, sending exception to main process:")
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	219 logger.debug(traceback.format_exc())
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	220 we = _get_worker_exception_data(wid)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	221 rep = (task_type, wid, [(task_data, (wid, we), False)])
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	222 put(rep)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	223 break
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	224
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	225 # Emergy abort.
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	226 elif task_type == _TASK_ABORT_WORKER:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	227 logger.debug("Worker %d got abort signal." % wid)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	228 break
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	229
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	230 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	231 raise Exception("Unknown task type: %s" % task_type)
693 d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	232
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	233 try:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	234 w.shutdown()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	235 except Exception as e:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	236 logger.error("Worker %s failed to shutdown.")
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	237 logger.exception(e)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	238 raise
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	239
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	240 logger.debug("Worker %d completed %d tasks." % (wid, completed))
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	241
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	242
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	243 class _WorkerParams:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	244 def __init__(self, wid, inqueue, outqueue, worker_class, initargs=(),
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	245 is_profiling=False, is_unit_testing=False):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	246 self.wid = wid
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	247 self.inqueue = inqueue
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	248 self.outqueue = outqueue
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	249 self.worker_class = worker_class
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	250 self.initargs = initargs
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	251 self.is_profiling = is_profiling
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	252 self.is_unit_testing = is_unit_testing
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	253
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	254
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	255 class WorkerPool:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	256 def __init__(self, worker_class, initargs=(), *,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	257 callback=None, error_callback=None,
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	258 worker_count=None, batch_size=None,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	259 userdata=None):
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	260 init_start_time = time.perf_counter()
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	261
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	262 stats = ExecutionStats()
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	263 stats.registerTimer('MasterInit')
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	264 self._stats = stats
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	265 self._time_in_put = 0
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	266 self._time_in_get = 0
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	267
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	268 self.userdata = userdata
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	269
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	270 worker_count = worker_count or os.cpu_count() or 1
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	271
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	272 if use_fastqueue:
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	273 self._task_queue = FastQueue()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	274 self._result_queue = FastQueue()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	275 self._quick_put = self._task_queue.put
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	276 self._quick_get = self._result_queue.get
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	277 else:
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	278 self._task_queue = multiprocessing.SimpleQueue()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	279 self._result_queue = multiprocessing.SimpleQueue()
878 313db67cfc35 bake: Use the basic Python queue again. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	280 self._quick_put = self._task_queue.put
313db67cfc35 bake: Use the basic Python queue again. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	281 self._quick_get = self._result_queue.get
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	282
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	283 self._callback = callback
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	284 self._error_callback = error_callback
462 04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 461 diff changeset	285 self._batch_size = batch_size
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	286 self._jobs_left = 0
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	287 self._lock_jobs_left = threading.Lock()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	288 self._lock_workers = threading.Lock()
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	289 self._event = threading.Event()
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	290 self._error_on_join = None
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	291 self._closed = False
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	292
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	293 main_module = sys.modules['__main__']
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	294 is_profiling = os.path.basename(main_module.__file__) in [
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	295 'profile.py', 'cProfile.py']
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	296 is_unit_testing = os.path.basename(main_module.__file__) in [
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	297 'py.test']
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	298
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	299 self._pool = []
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	300 for i in range(worker_count):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	301 worker_params = _WorkerParams(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	302 i, self._task_queue, self._result_queue,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	303 worker_class, initargs,
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	304 is_profiling=is_profiling,
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	305 is_unit_testing=is_unit_testing)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	306 w = multiprocessing.Process(target=worker_func,
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	307 args=(worker_params,))
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	308 w.name = w.name.replace('Process', 'PoolWorker')
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	309 w.daemon = True
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	310 w.start()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	311 self._pool.append(w)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	312
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	313 self._result_handler = threading.Thread(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	314 target=WorkerPool._handleResults,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	315 args=(self,))
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	316 self._result_handler.daemon = True
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	317 self._result_handler.start()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	318
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	319 stats.stepTimerSince('MasterInit', init_start_time)
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	320
1014 071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	321 @property
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	322 def pool_size(self):
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	323 return len(self._pool)
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	324
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	325 def queueJobs(self, jobs):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	326 if self._closed:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	327 if self._error_on_join:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	328 raise self._error_on_join
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	329 raise Exception("This worker pool has been closed.")
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	330
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	331 jobs = list(jobs)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	332 new_job_count = len(jobs)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	333 if new_job_count > 0:
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	334 put_start_time = time.perf_counter()
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	335
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	336 with self._lock_jobs_left:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	337 self._jobs_left += new_job_count
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	338
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	339 self._event.clear()
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	340 bs = self._batch_size
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	341 if not bs:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	342 for job in jobs:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	343 self._quick_put((TASK_JOB, job))
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	344 else:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	345 cur_offset = 0
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	346 while cur_offset < new_job_count:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	347 next_batch_idx = min(cur_offset + bs, new_job_count)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	348 job_batch = jobs[cur_offset:next_batch_idx]
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	349 self._quick_put((TASK_JOB_BATCH, job_batch))
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	350 cur_offset = next_batch_idx
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	351
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	352 self._time_in_put += (time.perf_counter() - put_start_time)
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	353 else:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	354 with self._lock_jobs_left:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	355 done = (self._jobs_left == 0)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	356 if done:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	357 self._event.set()
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	358
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	359 def wait(self, timeout=None):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	360 if self._closed:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	361 raise Exception("This worker pool has been closed.")
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	362
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	363 ret = self._event.wait(timeout)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	364 if self._error_on_join:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	365 raise self._error_on_join
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	366 return ret
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	367
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	368 def close(self):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	369 if self._closed:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	370 raise Exception("This worker pool has been closed.")
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	371 if self._jobs_left > 0:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	372 raise Exception("A previous job queue has not finished yet.")
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	373 if not self._event.is_set():
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	374 raise Exception("A previous job queue hasn't been cleared.")
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	375
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	376 close_start_time = time.perf_counter()
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	377 logger.debug("Closing worker pool...")
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	378 live_workers = list(filter(lambda w: w is not None, self._pool))
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	379 handler = _ReportHandler(len(live_workers))
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	380 self._callback = handler._handle
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	381 self._error_callback = handler._handleError
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	382 for w in live_workers:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	383 self._quick_put((TASK_END, None))
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	384 for w in live_workers:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	385 w.join()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	386
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	387 logger.debug("Waiting for reports...")
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	388 if not handler.wait(2):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	389 missing = handler.reports.index(None)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	390 logger.warning(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	391 "Didn't receive all worker reports before timeout. "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	392 "Missing report from worker %d." % missing)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	393
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	394 logger.debug("Exiting result handler thread...")
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	395 self._result_queue.put(None)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	396 self._result_handler.join()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	397 self._closed = True
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	398
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	399 stats = self._stats
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	400 stats.registerTimer('MasterTaskPut', time=self._time_in_put)
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	401 stats.registerTimer('MasterResultGet', time=self._time_in_get)
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	402 stats.registerTimer('MasterClose',
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	403 time=(time.perf_counter() - close_start_time))
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	404
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	405 return [stats] + handler.reports
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	406
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	407 def _onResultHandlerCriticalError(self, wid):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	408 logger.error("Result handler received a critical error from "
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	409 "worker %d." % wid)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	410 with self._lock_workers:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	411 self._pool[wid] = None
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	412 if all(map(lambda w: w is None, self._pool)):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	413 logger.error("All workers have died!")
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	414 self._closed = True
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	415 self._error_on_join = Exception("All workers have died!")
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	416 self._event.set()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	417 return False
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	418
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	419 return True
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	420
1008 09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	421 def _onTaskDone(self, done_count):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	422 with self._lock_jobs_left:
1008 09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	423 left = self._jobs_left - done_count
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	424 self._jobs_left = left
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	425
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	426 if left == 0:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	427 self._event.set()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	428
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	429 @staticmethod
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	430 def _handleResults(pool):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	431 userdata = pool.userdata
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	432 while True:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	433 try:
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	434 get_start_time = time.perf_counter()
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	435 res = pool._quick_get()
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	436 pool._time_in_get = (time.perf_counter() - get_start_time)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	437 except (EOFError, OSError):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	438 logger.debug("Result handler thread encountered connection "
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	439 "problem, exiting.")
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	440 return
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	441
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	442 if res is None:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	443 logger.debug("Result handler exiting.")
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	444 return
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	445
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	446 task_type, wid, res_data_list = res
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	447 for res_data in res_data_list:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	448 try:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	449 task_data, data, success = res_data
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	450 if success:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	451 if pool._callback:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	452 pool._callback(task_data, data, userdata)
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	453 else:
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	454 if task_type == _CRITICAL_WORKER_ERROR:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	455 logger.error(data)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	456 do_continue = pool._onResultHandlerCriticalError(wid)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	457 if not do_continue:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	458 logger.debug("Aborting result handling thread.")
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	459 return
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	460 else:
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	461 if pool._error_callback:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	462 pool._error_callback(task_data, data, userdata)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	463 else:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	464 logger.error(
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	465 "Worker %d failed to process a job:" % wid)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	466 logger.error(data)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	467 except Exception as ex:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	468 logger.exception(ex)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	469
1008 09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	470 if task_type == TASK_JOB or task_type == TASK_JOB_BATCH:
09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	471 pool._onTaskDone(len(res_data_list))
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	472
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	473
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	474 class _ReportHandler:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	475 def __init__(self, worker_count):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	476 self.reports = [None] * worker_count
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	477 self._count = worker_count
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	478 self._received = 0
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	479 self._lock = threading.Lock()
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	480 self._event = threading.Event()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	481
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	482 def wait(self, timeout=None):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	483 return self._event.wait(timeout)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	484
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	485 def _handle(self, job, res, _):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	486 wid, data = res
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	487 if wid < 0 or wid > self._count:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	488 logger.error("Ignoring report from unknown worker %d." % wid)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	489 return
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	490
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	491 stats = ExecutionStats()
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	492 stats.fromData(data)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	493
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	494 with self._lock:
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	495 self.reports[wid] = stats
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	496 self._received += 1
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	497 if self._received == self._count:
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	498 self._event.set()
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	499
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	500 def _handleError(self, job, res, _):
1019 bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	501 logger.error("Worker %d failed to send its report." % res[0])
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	502 logger.error(res)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	503
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	504
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	505 class FastQueue:
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	506 def __init__(self):
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	507 self._reader, self._writer = multiprocessing.Pipe(duplex=False)
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	508 self._rlock = multiprocessing.Lock()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	509 self._wlock = multiprocessing.Lock()
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	510 self._initBuffers()
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	511 self._initSerializer()
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	512
9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	513 def _initBuffers(self):
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	514 self._rbuf = io.BytesIO()
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	515 self._rbuf.truncate(256)
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	516 self._wbuf = io.BytesIO()
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	517 self._wbuf.truncate(256)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	518
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	519 def _initSerializer(self):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	520 pass
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	521
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	522 def __getstate__(self):
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	523 return (self._reader, self._writer, self._rlock, self._wlock)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	524
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	525 def __setstate__(self, state):
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	526 (self._reader, self._writer, self._rlock, self._wlock) = state
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	527 self._initBuffers()
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	528
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	529 def get(self):
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	530 with self._rlock:
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	531 self._rbuf.seek(0)
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	532 try:
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	533 with self._rbuf.getbuffer() as b:
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	534 bufsize = self._reader.recv_bytes_into(b)
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	535 except multiprocessing.BufferTooShort as e:
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	536 bufsize = len(e.args[0])
9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	537 self._rbuf.truncate(bufsize * 2)
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	538 self._rbuf.seek(0)
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	539 self._rbuf.write(e.args[0])
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	540
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	541 self._rbuf.seek(0)
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	542 return _unpickle(self, self._rbuf, bufsize)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	543
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	544 def put(self, obj):
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	545 self._wbuf.seek(0)
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	546 _pickle(self, obj, self._wbuf)
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	547 size = self._wbuf.tell()
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	548
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	549 self._wbuf.seek(0)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	550 with self._wlock:
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	551 with self._wbuf.getbuffer() as b:
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	552 self._writer.send_bytes(b, 0, size)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	553
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	554
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	555 class _BufferWrapper:
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	556 def __init__(self, buf, read_size=0):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	557 self._buf = buf
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	558 self._read_size = read_size
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	559
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	560 def write(self, data):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	561 self._buf.write(data.encode('utf8'))
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	562
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	563 def read(self):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	564 return self._buf.read(self._read_size).decode('utf8')
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	565
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	566
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	567 if use_fastpickle:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	568 from piecrust import fastpickle
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	569
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	570 def _pickle_fast(queue, obj, buf):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	571 fastpickle.pickle_intob(obj, buf)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	572
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	573 def _unpickle_fast(queue, buf, bufsize):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	574 return fastpickle.unpickle_fromb(buf, bufsize)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	575
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	576 _pickle = _pickle_fast
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	577 _unpickle = _unpickle_fast
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	578
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	579 elif use_msgpack:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	580 import msgpack
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	581
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	582 def _pickle_msgpack(queue, obj, buf):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	583 buf.write(queue._packer.pack(obj))
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	584
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	585 def _unpickle_msgpack(queue, buf, bufsize):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	586 queue._unpacker.feed(buf.getbuffer())
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	587 for o in queue._unpacker:
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	588 return o
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	589 # return msgpack.unpack(buf)
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	590
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	591 def _init_msgpack(queue):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	592 queue._packer = msgpack.Packer()
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	593 queue._unpacker = msgpack.Unpacker()
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	594
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	595 _pickle = _pickle_msgpack
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	596 _unpickle = _unpickle_msgpack
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	597 FastQueue._initSerializer = _init_msgpack
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	598
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	599 elif use_marshall:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	600 import marshal
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	601
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	602 def _pickle_marshal(queue, obj, buf):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	603 marshal.dump(obj, buf)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	604
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	605 def _unpickle_marshal(queue, buf, bufsize):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	606 return marshal.load(buf)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	607
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	608 _pickle = _pickle_marshal
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	609 _unpickle = _unpickle_marshal
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	610
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	611 elif use_json:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	612 import json
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	613
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	614 def _pickle_json(queue, obj, buf):
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	615 buf = _BufferWrapper(buf)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	616 json.dump(obj, buf, indent=None, separators=(',', ':'))
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	617
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	618 def _unpickle_json(queue, buf, bufsize):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	619 buf = _BufferWrapper(buf, bufsize)
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	620 return json.load(buf)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	621
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	622 _pickle = _pickle_json
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	623 _unpickle = _unpickle_json
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	624
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	625 else:
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	626 import pickle
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	627
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	628 def _pickle_default(queue, obj, buf):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	629 pickle.dump(obj, buf, pickle.HIGHEST_PROTOCOL)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	630
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	631 def _unpickle_default(queue, buf, bufsize):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	632 return pickle.load(buf)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	633
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	634 _pickle = _pickle_default
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	635 _unpickle = _unpickle_default

Mercurial > piecrust2

annotate piecrust/workerpool.py @ 1188:a7c43131d871