piecrust2: piecrust/workerpool.py annotate

annotate piecrust/workerpool.py @ 1095:c8518f5cedbb

serve: Do some more useful debug logging when serving assets.

author	Ludovic Chabant <ludovic@chabant.com>
date	Sat, 17 Feb 2018 11:50:42 -0800
parents	aa4f1e04cf3e
children

rev	line source
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	1 import io
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import sys
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	4 import time
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import logging
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 import threading
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	7 import traceback
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8 import multiprocessing
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	9 from piecrust.environment import ExecutionStats
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	10
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	11
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12 logger = logging.getLogger(__name__)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	14 use_fastqueue = False
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	15 use_fastpickle = False
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	16 use_msgpack = False
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	17 use_marshall = False
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	18 use_json = False
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	19
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21 class IWorker(object):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	22 """ Interface for a pool worker.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	23 """
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24 def initialize(self):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	25 raise NotImplementedError()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27 def process(self, job):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 raise NotImplementedError()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	30 def getStats(self):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31 return None
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32
702 c62d83e17abf bake: Some more optimizations. Ludovic Chabant <ludovic@chabant.com> parents: 697 diff changeset	33 def shutdown(self):
c62d83e17abf bake: Some more optimizations. Ludovic Chabant <ludovic@chabant.com> parents: 697 diff changeset	34 pass
c62d83e17abf bake: Some more optimizations. Ludovic Chabant <ludovic@chabant.com> parents: 697 diff changeset	35
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	37 def _get_worker_exception_data(wid):
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	38 t, v, tb = sys.exc_info()
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	39 return {
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	40 'wid': wid,
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	41 'type': str(t),
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	42 'value': '\n'.join(_get_errors(v)),
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	43 'traceback': ''.join(traceback.format_exception(t, v, tb))
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	44 }
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	45
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	46
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	47 def _get_errors(ex):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	48 errors = []
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	49 while ex is not None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	50 msg = str(ex)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	51 errors.append(msg)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	52 ex = ex.__cause__
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	53 return errors
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	54
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	55
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	56 TASK_JOB = 0
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	57 TASK_JOB_BATCH = 1
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	58 TASK_END = 2
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	59 _TASK_ABORT_WORKER = 10
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	60 _CRITICAL_WORKER_ERROR = 11
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	61
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	62
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63 def worker_func(params):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64 if params.is_profiling:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	65 try:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	66 import cProfile as profile
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	67 except ImportError:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 import profile
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	70 params.is_profiling = False
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71 name = params.worker_class.__name__
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	72 profile.runctx('_real_worker_func(params)',
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	73 globals(), locals(),
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	74 filename='%s-%d.prof' % (name, params.wid))
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75 else:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76 _real_worker_func(params)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	77
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	78
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	79 def _real_worker_func(params):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	80 try:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	81 _real_worker_func_unsafe(params)
1046 aa4f1e04cf3e serve: Don't spam the output when hitting CTRL+C while processing assets. Ludovic Chabant <ludovic@chabant.com> parents: 1019 diff changeset	82 except (KeyboardInterrupt, SystemExit):
aa4f1e04cf3e serve: Don't spam the output when hitting CTRL+C while processing assets. Ludovic Chabant <ludovic@chabant.com> parents: 1019 diff changeset	83 # Return silently
aa4f1e04cf3e serve: Don't spam the output when hitting CTRL+C while processing assets. Ludovic Chabant <ludovic@chabant.com> parents: 1019 diff changeset	84 pass
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	85 except Exception as ex:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	86 logger.exception(ex)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	87 msg = ("CRITICAL ERROR IN WORKER %d\n%s" % (params.wid, str(ex)))
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	88 params.outqueue.put((
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	89 _CRITICAL_WORKER_ERROR, None, False, params.wid, msg))
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	90
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	91
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	92 def _pre_parse_pytest_args():
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	93 # If we are unit-testing, we need to translate our test logging
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	94 # arguments into something Chef can understand.
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	95 import argparse
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	96 parser = argparse.ArgumentParser()
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	97 # This is adapted from our `conftest.py`.
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	98 parser.add_argument('--log-debug', action='store_true')
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	99 parser.add_argument('--log-file')
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	100 res, _ = parser.parse_known_args(sys.argv[1:])
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	101
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	102 chef_args = []
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	103 if res.log_debug:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	104 chef_args.append('--debug')
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	105 if res.log_file:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	106 chef_args += ['--log', res.log_file]
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	107
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	108 root_logger = logging.getLogger()
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	109 while len(root_logger.handlers) > 0:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	110 root_logger.removeHandler(root_logger.handlers[0])
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	111
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	112 from piecrust.main import _pre_parse_chef_args
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	113 _pre_parse_chef_args(chef_args)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	114
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	115
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	116 def _real_worker_func_unsafe(params):
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	117 init_start_time = time.perf_counter()
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	118
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	119 wid = params.wid
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	120
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	121 stats = ExecutionStats()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	122 stats.registerTimer('WorkerInit')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	123
500 22a230d99621 bake: Fix logging configuration for multi-processing on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	124 # In a context where `multiprocessing` is using the `spawn` forking model,
22a230d99621 bake: Fix logging configuration for multi-processing on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	125 # the new process doesn't inherit anything, so we lost all our logging
22a230d99621 bake: Fix logging configuration for multi-processing on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	126 # configuration here. Let's set it up again.
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	127 if (hasattr(multiprocessing, 'get_start_method') and
566 8073ae8cb164 bake: Don't re-setup logging for workers unless we're sure we need it. Ludovic Chabant <ludovic@chabant.com> parents: 500 diff changeset	128 multiprocessing.get_start_method() == 'spawn'):
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	129 if not params.is_unit_testing:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	130 from piecrust.main import _pre_parse_chef_args
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	131 _pre_parse_chef_args(sys.argv[1:])
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	132 else:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	133 _pre_parse_pytest_args()
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	134 elif params.is_unit_testing:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	135 _pre_parse_pytest_args()
500 22a230d99621 bake: Fix logging configuration for multi-processing on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	136
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	137 from piecrust.main import ColoredFormatter
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	138 root_logger = logging.getLogger()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	139 root_logger.handlers[0].setFormatter(ColoredFormatter(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	140 ('[W-%d]' % wid) + '[%(name)s] %(message)s'))
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	141
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	142 logger.debug("Worker %d initializing..." % wid)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	143
693 d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	144 # We don't need those.
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	145 params.inqueue._writer.close()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	146 params.outqueue._reader.close()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	147
693 d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	148 # Initialize the underlying worker class.
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	149 w = params.worker_class(*params.initargs)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150 w.wid = wid
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	151 try:
8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	152 w.initialize()
8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	153 except Exception as ex:
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	154 logger.error("Worker %d failed to initialize." % wid)
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	155 logger.exception(ex)
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	156 raise
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	157
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	158 stats.stepTimerSince('WorkerInit', init_start_time)
693 d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	159
d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	160 # Start pumping!
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	161 completed = 0
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	162 time_in_get = 0
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	163 time_in_put = 0
1019 bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	164 is_first_get = True
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	165 get = params.inqueue.get
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	166 put = params.outqueue.put
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	167
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	168 while True:
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	169 get_start_time = time.perf_counter()
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	170 task = get()
1019 bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	171 if not is_first_get:
bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	172 time_in_get += (time.perf_counter() - get_start_time)
bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	173 else:
bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	174 is_first_get = False
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	175
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	176 task_type, task_data = task
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	177
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	178 # Job task(s)... just do it.
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	179 if task_type == TASK_JOB or task_type == TASK_JOB_BATCH:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	180
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	181 task_data_list = task_data
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	182 if task_type == TASK_JOB:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	183 task_data_list = [task_data]
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	184
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	185 result_list = []
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	186
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	187 for td in task_data_list:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	188 try:
1008 09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	189 worker_res = w.process(td)
09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	190 result_list.append((td, worker_res, True))
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	191 except Exception as e:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	192 logger.debug(
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	193 "Error processing job, sending exception to main process:")
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	194 logger.debug(traceback.format_exc())
1019 bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	195 error_res = _get_worker_exception_data(wid)
1008 09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	196 result_list.append((td, error_res, False))
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	197
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	198 res = (task_type, wid, result_list)
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	199 put_start_time = time.perf_counter()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	200 put(res)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	201 time_in_put += (time.perf_counter() - put_start_time)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	202
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	203 completed += len(task_data_list)
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	204
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	205 # End task... gather stats to send back to the main process.
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	206 elif task_type == TASK_END:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	207 logger.debug("Worker %d got end task, exiting." % wid)
1019 bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	208 stats.registerTimer('Worker_%d_TaskGet' % wid, time=time_in_get)
bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	209 stats.registerTimer('Worker_all_TaskGet', time=time_in_get)
bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	210 stats.registerTimer('Worker_%d_ResultPut' % wid, time=time_in_put)
bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	211 stats.registerTimer('Worker_all_ResultPut', time=time_in_put)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	212 try:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	213 stats.mergeStats(w.getStats())
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	214 stats_data = stats.toData()
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	215 rep = (task_type, wid, [(task_data, (wid, stats_data), True)])
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	216 except Exception as e:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	217 logger.debug(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	218 "Error getting report, sending exception to main process:")
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	219 logger.debug(traceback.format_exc())
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	220 we = _get_worker_exception_data(wid)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	221 rep = (task_type, wid, [(task_data, (wid, we), False)])
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	222 put(rep)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	223 break
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	224
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	225 # Emergy abort.
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	226 elif task_type == _TASK_ABORT_WORKER:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	227 logger.debug("Worker %d got abort signal." % wid)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	228 break
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	229
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	230 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	231 raise Exception("Unknown task type: %s" % task_type)
693 d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	232
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	233 try:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	234 w.shutdown()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	235 except Exception as e:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	236 logger.error("Worker %s failed to shutdown.")
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	237 logger.exception(e)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	238 raise
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	239
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	240 logger.debug("Worker %d completed %d tasks." % (wid, completed))
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	241
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	242
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	243 class _WorkerParams:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	244 def __init__(self, wid, inqueue, outqueue, worker_class, initargs=(),
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	245 is_profiling=False, is_unit_testing=False):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	246 self.wid = wid
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	247 self.inqueue = inqueue
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	248 self.outqueue = outqueue
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	249 self.worker_class = worker_class
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	250 self.initargs = initargs
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	251 self.is_profiling = is_profiling
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	252 self.is_unit_testing = is_unit_testing
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	253
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	254
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	255 class WorkerPool:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	256 def __init__(self, worker_class, initargs=(), *,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	257 callback=None, error_callback=None,
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	258 worker_count=None, batch_size=None,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	259 userdata=None):
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	260 init_start_time = time.perf_counter()
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	261
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	262 stats = ExecutionStats()
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	263 stats.registerTimer('MasterInit')
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	264 self._stats = stats
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	265 self._time_in_put = 0
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	266 self._time_in_get = 0
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	267
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	268 self.userdata = userdata
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	269
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	270 worker_count = worker_count or os.cpu_count() or 1
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	271
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	272 if use_fastqueue:
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	273 self._task_queue = FastQueue()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	274 self._result_queue = FastQueue()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	275 self._quick_put = self._task_queue.put
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	276 self._quick_get = self._result_queue.get
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	277 else:
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	278 self._task_queue = multiprocessing.SimpleQueue()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	279 self._result_queue = multiprocessing.SimpleQueue()
878 313db67cfc35 bake: Use the basic Python queue again. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	280 self._quick_put = self._task_queue.put
313db67cfc35 bake: Use the basic Python queue again. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	281 self._quick_get = self._result_queue.get
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	282
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	283 self._callback = callback
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	284 self._error_callback = error_callback
462 04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 461 diff changeset	285 self._batch_size = batch_size
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	286 self._jobs_left = 0
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	287 self._lock_jobs_left = threading.Lock()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	288 self._lock_workers = threading.Lock()
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	289 self._event = threading.Event()
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	290 self._error_on_join = None
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	291 self._closed = False
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	292
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	293 main_module = sys.modules['__main__']
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	294 is_profiling = os.path.basename(main_module.__file__) in [
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	295 'profile.py', 'cProfile.py']
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	296 is_unit_testing = os.path.basename(main_module.__file__) in [
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	297 'py.test']
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	298
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	299 self._pool = []
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	300 for i in range(worker_count):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	301 worker_params = _WorkerParams(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	302 i, self._task_queue, self._result_queue,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	303 worker_class, initargs,
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	304 is_profiling=is_profiling,
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	305 is_unit_testing=is_unit_testing)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	306 w = multiprocessing.Process(target=worker_func,
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	307 args=(worker_params,))
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	308 w.name = w.name.replace('Process', 'PoolWorker')
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	309 w.daemon = True
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	310 w.start()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	311 self._pool.append(w)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	312
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	313 self._result_handler = threading.Thread(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	314 target=WorkerPool._handleResults,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	315 args=(self,))
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	316 self._result_handler.daemon = True
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	317 self._result_handler.start()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	318
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	319 stats.stepTimerSince('MasterInit', init_start_time)
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	320
1014 071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	321 @property
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	322 def pool_size(self):
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	323 return len(self._pool)
071f30aa04bb bake: Do template caching in a background job if possible. Ludovic Chabant <ludovic@chabant.com> parents: 1011 diff changeset	324
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	325 def queueJobs(self, jobs):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	326 if self._closed:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	327 if self._error_on_join:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	328 raise self._error_on_join
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	329 raise Exception("This worker pool has been closed.")
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	330
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	331 jobs = list(jobs)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	332 new_job_count = len(jobs)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	333 if new_job_count > 0:
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	334 put_start_time = time.perf_counter()
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	335
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	336 with self._lock_jobs_left:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	337 self._jobs_left += new_job_count
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	338
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	339 self._event.clear()
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	340 bs = self._batch_size
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	341 if not bs:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	342 for job in jobs:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	343 self._quick_put((TASK_JOB, job))
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	344 else:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	345 cur_offset = 0
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	346 while cur_offset < new_job_count:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	347 next_batch_idx = min(cur_offset + bs, new_job_count)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	348 job_batch = jobs[cur_offset:next_batch_idx]
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	349 self._quick_put((TASK_JOB_BATCH, job_batch))
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	350 cur_offset = next_batch_idx
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	351
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	352 self._time_in_put += (time.perf_counter() - put_start_time)
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	353 else:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	354 with self._lock_jobs_left:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	355 done = (self._jobs_left == 0)
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	356 if done:
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	357 self._event.set()
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	358
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	359 def wait(self, timeout=None):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	360 if self._closed:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	361 raise Exception("This worker pool has been closed.")
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	362
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	363 ret = self._event.wait(timeout)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	364 if self._error_on_join:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	365 raise self._error_on_join
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	366 return ret
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	367
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	368 def close(self):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	369 if self._closed:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	370 raise Exception("This worker pool has been closed.")
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	371 if self._jobs_left > 0:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	372 raise Exception("A previous job queue has not finished yet.")
979 45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	373 if not self._event.is_set():
45ad976712ec tests: Big push to get the tests to pass again. Ludovic Chabant <ludovic@chabant.com> parents: 971 diff changeset	374 raise Exception("A previous job queue hasn't been cleared.")
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	375
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	376 close_start_time = time.perf_counter()
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	377 logger.debug("Closing worker pool...")
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	378 live_workers = list(filter(lambda w: w is not None, self._pool))
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	379 handler = _ReportHandler(len(live_workers))
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	380 self._callback = handler._handle
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	381 self._error_callback = handler._handleError
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	382 for w in live_workers:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	383 self._quick_put((TASK_END, None))
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	384 for w in live_workers:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	385 w.join()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	386
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	387 logger.debug("Waiting for reports...")
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	388 if not handler.wait(2):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	389 missing = handler.reports.index(None)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	390 logger.warning(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	391 "Didn't receive all worker reports before timeout. "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	392 "Missing report from worker %d." % missing)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	393
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	394 logger.debug("Exiting result handler thread...")
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	395 self._result_queue.put(None)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	396 self._result_handler.join()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	397 self._closed = True
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	398
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	399 stats = self._stats
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	400 stats.registerTimer('MasterTaskPut', time=self._time_in_put)
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	401 stats.registerTimer('MasterResultGet', time=self._time_in_get)
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	402 stats.registerTimer('MasterClose',
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	403 time=(time.perf_counter() - close_start_time))
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	404
c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	405 return [stats] + handler.reports
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	406
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	407 def _onResultHandlerCriticalError(self, wid):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	408 logger.error("Result handler received a critical error from "
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	409 "worker %d." % wid)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	410 with self._lock_workers:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	411 self._pool[wid] = None
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	412 if all(map(lambda w: w is None, self._pool)):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	413 logger.error("All workers have died!")
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	414 self._closed = True
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	415 self._error_on_join = Exception("All workers have died!")
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	416 self._event.set()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	417 return False
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	418
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	419 return True
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	420
1008 09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	421 def _onTaskDone(self, done_count):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	422 with self._lock_jobs_left:
1008 09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	423 left = self._jobs_left - done_count
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	424 self._jobs_left = left
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	425
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	426 if left == 0:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	427 self._event.set()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	428
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	429 @staticmethod
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	430 def _handleResults(pool):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	431 userdata = pool.userdata
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	432 while True:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	433 try:
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	434 get_start_time = time.perf_counter()
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	435 res = pool._quick_get()
1011 c4cf3cfe2726 bake: Better performance stats, and add callback to preload templates. Ludovic Chabant <ludovic@chabant.com> parents: 1008 diff changeset	436 pool._time_in_get = (time.perf_counter() - get_start_time)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	437 except (EOFError, OSError):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	438 logger.debug("Result handler thread encountered connection "
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	439 "problem, exiting.")
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	440 return
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	441
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	442 if res is None:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	443 logger.debug("Result handler exiting.")
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	444 return
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	445
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	446 task_type, wid, res_data_list = res
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	447 for res_data in res_data_list:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	448 try:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	449 task_data, data, success = res_data
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	450 if success:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	451 if pool._callback:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	452 pool._callback(task_data, data, userdata)
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	453 else:
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	454 if task_type == _CRITICAL_WORKER_ERROR:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	455 logger.error(data)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	456 do_continue = pool._onResultHandlerCriticalError(wid)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	457 if not do_continue:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	458 logger.debug("Aborting result handling thread.")
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	459 return
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	460 else:
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	461 if pool._error_callback:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	462 pool._error_callback(task_data, data, userdata)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	463 else:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	464 logger.error(
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	465 "Worker %d failed to process a job:" % wid)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	466 logger.error(data)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	467 except Exception as ex:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	468 logger.exception(ex)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	469
1008 09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	470 if task_type == TASK_JOB or task_type == TASK_JOB_BATCH:
09c3d415d9e5 bake: Make `batch-size` option work again. Ludovic Chabant <ludovic@chabant.com> parents: 991 diff changeset	471 pool._onTaskDone(len(res_data_list))
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	472
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	473
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	474 class _ReportHandler:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	475 def __init__(self, worker_count):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	476 self.reports = [None] * worker_count
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	477 self._count = worker_count
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	478 self._received = 0
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	479 self._lock = threading.Lock()
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	480 self._event = threading.Event()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	481
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	482 def wait(self, timeout=None):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	483 return self._event.wait(timeout)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	484
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	485 def _handle(self, job, res, _):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	486 wid, data = res
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	487 if wid < 0 or wid > self._count:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	488 logger.error("Ignoring report from unknown worker %d." % wid)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	489 return
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	490
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	491 stats = ExecutionStats()
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	492 stats.fromData(data)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	493
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	494 with self._lock:
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	495 self.reports[wid] = stats
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	496 self._received += 1
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	497 if self._received == self._count:
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	498 self._event.set()
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	499
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	500 def _handleError(self, job, res, _):
1019 bd544b65cfad bake: More detailed stats, and fix a problem with some error reporting. Ludovic Chabant <ludovic@chabant.com> parents: 1014 diff changeset	501 logger.error("Worker %d failed to send its report." % res[0])
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	502 logger.error(res)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	503
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	504
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	505 class FastQueue:
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	506 def __init__(self):
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	507 self._reader, self._writer = multiprocessing.Pipe(duplex=False)
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	508 self._rlock = multiprocessing.Lock()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	509 self._wlock = multiprocessing.Lock()
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	510 self._initBuffers()
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	511 self._initSerializer()
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	512
9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	513 def _initBuffers(self):
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	514 self._rbuf = io.BytesIO()
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	515 self._rbuf.truncate(256)
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	516 self._wbuf = io.BytesIO()
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	517 self._wbuf.truncate(256)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	518
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	519 def _initSerializer(self):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	520 pass
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	521
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	522 def __getstate__(self):
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	523 return (self._reader, self._writer, self._rlock, self._wlock)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	524
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	525 def __setstate__(self, state):
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	526 (self._reader, self._writer, self._rlock, self._wlock) = state
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	527 self._initBuffers()
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	528
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	529 def get(self):
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	530 with self._rlock:
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	531 self._rbuf.seek(0)
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	532 try:
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	533 with self._rbuf.getbuffer() as b:
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	534 bufsize = self._reader.recv_bytes_into(b)
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	535 except multiprocessing.BufferTooShort as e:
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	536 bufsize = len(e.args[0])
9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	537 self._rbuf.truncate(bufsize * 2)
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	538 self._rbuf.seek(0)
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	539 self._rbuf.write(e.args[0])
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	540
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	541 self._rbuf.seek(0)
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	542 return _unpickle(self, self._rbuf, bufsize)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	543
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	544 def put(self, obj):
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	545 self._wbuf.seek(0)
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	546 _pickle(self, obj, self._wbuf)
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	547 size = self._wbuf.tell()
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	548
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	549 self._wbuf.seek(0)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	550 with self._wlock:
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	551 with self._wbuf.getbuffer() as b:
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	552 self._writer.send_bytes(b, 0, size)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	553
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	554
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	555 class _BufferWrapper:
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	556 def __init__(self, buf, read_size=0):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	557 self._buf = buf
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	558 self._read_size = read_size
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	559
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	560 def write(self, data):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	561 self._buf.write(data.encode('utf8'))
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	562
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	563 def read(self):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	564 return self._buf.read(self._read_size).decode('utf8')
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	565
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	566
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	567 if use_fastpickle:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	568 from piecrust import fastpickle
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	569
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	570 def _pickle_fast(queue, obj, buf):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	571 fastpickle.pickle_intob(obj, buf)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	572
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	573 def _unpickle_fast(queue, buf, bufsize):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	574 return fastpickle.unpickle_fromb(buf, bufsize)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	575
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	576 _pickle = _pickle_fast
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	577 _unpickle = _unpickle_fast
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	578
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	579 elif use_msgpack:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	580 import msgpack
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	581
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	582 def _pickle_msgpack(queue, obj, buf):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	583 buf.write(queue._packer.pack(obj))
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	584
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	585 def _unpickle_msgpack(queue, buf, bufsize):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	586 queue._unpacker.feed(buf.getbuffer())
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	587 for o in queue._unpacker:
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	588 return o
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	589 # return msgpack.unpack(buf)
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	590
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	591 def _init_msgpack(queue):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	592 queue._packer = msgpack.Packer()
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	593 queue._unpacker = msgpack.Unpacker()
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	594
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	595 _pickle = _pickle_msgpack
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	596 _unpickle = _unpickle_msgpack
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	597 FastQueue._initSerializer = _init_msgpack
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	598
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	599 elif use_marshall:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	600 import marshal
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	601
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	602 def _pickle_marshal(queue, obj, buf):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	603 marshal.dump(obj, buf)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	604
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	605 def _unpickle_marshal(queue, buf, bufsize):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	606 return marshal.load(buf)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	607
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	608 _pickle = _pickle_marshal
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	609 _unpickle = _unpickle_marshal
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	610
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	611 elif use_json:
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	612 import json
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	613
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	614 def _pickle_json(queue, obj, buf):
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	615 buf = _BufferWrapper(buf)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	616 json.dump(obj, buf, indent=None, separators=(',', ':'))
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	617
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	618 def _unpickle_json(queue, buf, bufsize):
1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	619 buf = _BufferWrapper(buf, bufsize)
989 8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	620 return json.load(buf)
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	621
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	622 _pickle = _pickle_json
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	623 _unpickle = _unpickle_json
8adc27285d93 bake: Big pass on bake performance. Ludovic Chabant <ludovic@chabant.com> parents: 979 diff changeset	624
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	625 else:
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	626 import pickle
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	627
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	628 def _pickle_default(queue, obj, buf):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	629 pickle.dump(obj, buf, pickle.HIGHEST_PROTOCOL)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	630
991 1857dbd4580f bake: Fix bugs introduced by bake optimizations, of course. Ludovic Chabant <ludovic@chabant.com> parents: 989 diff changeset	631 def _unpickle_default(queue, buf, bufsize):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	632 return pickle.load(buf)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	633
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	634 _pickle = _pickle_default
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	635 _unpickle = _unpickle_default

Mercurial > piecrust2

annotate piecrust/workerpool.py @ 1095:c8518f5cedbb