piecrust2: piecrust/workerpool.py annotate

annotate piecrust/workerpool.py @ 978:7e51d14097cb

config: Properly pass the merge mode to the `merge_dicts` function.

author	Ludovic Chabant <ludovic@chabant.com>
date	Sun, 29 Oct 2017 22:46:41 -0700
parents	5485a11591ec
children	45ad976712ec

rev	line source
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	1 import io
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	2 import os
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	3 import sys
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	4 import time
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	5 import logging
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	6 import threading
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	7 import traceback
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	8 import multiprocessing
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	9 from piecrust.environment import ExecutionStats
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	10
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	11
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	12 logger = logging.getLogger(__name__)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	13
878 313db67cfc35 bake: Use the basic Python queue again. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	14 use_fastqueue = False
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	15
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	16 use_fastpickle = False
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	17 use_msgpack = False
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	18 use_marshall = False
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	19
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	20
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	21 class IWorker(object):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	22 """ Interface for a pool worker.
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	23 """
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	24 def initialize(self):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	25 raise NotImplementedError()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	26
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	27 def process(self, job):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	28 raise NotImplementedError()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	29
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	30 def getStats(self):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	31 return None
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	32
702 c62d83e17abf bake: Some more optimizations. Ludovic Chabant <ludovic@chabant.com> parents: 697 diff changeset	33 def shutdown(self):
c62d83e17abf bake: Some more optimizations. Ludovic Chabant <ludovic@chabant.com> parents: 697 diff changeset	34 pass
c62d83e17abf bake: Some more optimizations. Ludovic Chabant <ludovic@chabant.com> parents: 697 diff changeset	35
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	36
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	37 class WorkerExceptionData:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	38 def __init__(self, wid):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	39 super().__init__()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	40 self.wid = wid
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	41 t, v, tb = sys.exc_info()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	42 self.type = t
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	43 self.value = '\n'.join(_get_errors(v))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	44 self.traceback = ''.join(traceback.format_exception(t, v, tb))
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	45
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	46 def __str__(self):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	47 return str(self.value)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	48
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	49
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	50 def _get_errors(ex):
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	51 errors = []
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	52 while ex is not None:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	53 msg = str(ex)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	54 errors.append(msg)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	55 ex = ex.__cause__
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	56 return errors
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	57
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	58
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	59 TASK_JOB = 0
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	60 TASK_END = 1
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	61 _TASK_ABORT_WORKER = 10
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	62 _CRITICAL_WORKER_ERROR = 11
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	63
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	64
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	65 def worker_func(params):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	66 if params.is_profiling:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	67 try:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	68 import cProfile as profile
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	69 except ImportError:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	70 import profile
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	71
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	72 params.is_profiling = False
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	73 name = params.worker_class.__name__
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	74 profile.runctx('_real_worker_func(params)',
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	75 globals(), locals(),
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	76 filename='%s-%d.prof' % (name, params.wid))
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	77 else:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	78 _real_worker_func(params)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	79
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	80
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	81 def _real_worker_func(params):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	82 try:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	83 _real_worker_func_unsafe(params)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	84 except Exception as ex:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	85 logger.exception(ex)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	86 msg = ("CRITICAL ERROR IN WORKER %d\n%s" % (params.wid, str(ex)))
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	87 params.outqueue.put((
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	88 _CRITICAL_WORKER_ERROR, None, False, params.wid, msg))
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	89
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	90
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	91 def _real_worker_func_unsafe(params):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	92 wid = params.wid
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	93
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	94 stats = ExecutionStats()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	95 stats.registerTimer('WorkerInit')
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	96 init_start_time = time.perf_counter()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	97
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	98 # If we are unit-testing, we didn't setup all the logging environment
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	99 # yet, since the executable is `py.test`. We need to translate our
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	100 # test logging arguments into something Chef can understand.
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	101 if params.is_unit_testing:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	102 import argparse
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	103 parser = argparse.ArgumentParser()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	104 # This is adapted from our `conftest.py`.
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	105 parser.add_argument('--log-debug', action='store_true')
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	106 parser.add_argument('--log-file')
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	107 res, _ = parser.parse_known_args(sys.argv[1:])
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	108
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	109 chef_args = []
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	110 if res.log_debug:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	111 chef_args.append('--debug')
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	112 if res.log_file:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	113 chef_args += ['--log', res.log_file]
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	114
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	115 from piecrust.main import _pre_parse_chef_args
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	116 _pre_parse_chef_args(chef_args)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	117
500 22a230d99621 bake: Fix logging configuration for multi-processing on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	118 # In a context where `multiprocessing` is using the `spawn` forking model,
22a230d99621 bake: Fix logging configuration for multi-processing on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	119 # the new process doesn't inherit anything, so we lost all our logging
22a230d99621 bake: Fix logging configuration for multi-processing on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	120 # configuration here. Let's set it up again.
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	121 elif (hasattr(multiprocessing, 'get_start_method') and
566 8073ae8cb164 bake: Don't re-setup logging for workers unless we're sure we need it. Ludovic Chabant <ludovic@chabant.com> parents: 500 diff changeset	122 multiprocessing.get_start_method() == 'spawn'):
8073ae8cb164 bake: Don't re-setup logging for workers unless we're sure we need it. Ludovic Chabant <ludovic@chabant.com> parents: 500 diff changeset	123 from piecrust.main import _pre_parse_chef_args
8073ae8cb164 bake: Don't re-setup logging for workers unless we're sure we need it. Ludovic Chabant <ludovic@chabant.com> parents: 500 diff changeset	124 _pre_parse_chef_args(sys.argv[1:])
500 22a230d99621 bake: Fix logging configuration for multi-processing on Windows. Ludovic Chabant <ludovic@chabant.com> parents: 462 diff changeset	125
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	126 from piecrust.main import ColoredFormatter
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	127 root_logger = logging.getLogger()
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	128 root_logger.handlers[0].setFormatter(ColoredFormatter(
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	129 ('[W-%d]' % wid) + '[%(name)s] %(message)s'))
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	130
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	131 logger.debug("Worker %d initializing..." % wid)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	132
693 d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	133 # We don't need those.
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	134 params.inqueue._writer.close()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	135 params.outqueue._reader.close()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	136
693 d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	137 # Initialize the underlying worker class.
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	138 w = params.worker_class(*params.initargs)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	139 w.wid = wid
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	140 try:
8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	141 w.initialize()
8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	142 except Exception as ex:
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	143 logger.error("Worker %d failed to initialize." % wid)
453 8351a77e13f5 bake: Don't pass the previous record entries to the workers. Ludovic Chabant <ludovic@chabant.com> parents: 451 diff changeset	144 logger.exception(ex)
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	145 raise
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	146
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	147 stats.stepTimerSince('WorkerInit', init_start_time)
693 d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	148
d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	149 # Start pumping!
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	150 completed = 0
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	151 time_in_get = 0
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	152 time_in_put = 0
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	153 get = params.inqueue.get
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	154 put = params.outqueue.put
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	155
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	156 while True:
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	157 get_start_time = time.perf_counter()
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	158 task = get()
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	159 time_in_get += (time.perf_counter() - get_start_time)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	160
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	161 task_type, task_data = task
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	162
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	163 # Job task... just do it.
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	164 if task_type == TASK_JOB:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	165 try:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	166 res = (task_type, task_data, True, wid, w.process(task_data))
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	167 except Exception as e:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	168 logger.debug(
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	169 "Error processing job, sending exception to main process:")
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	170 logger.debug(traceback.format_exc())
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	171 we = WorkerExceptionData(wid)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	172 res = (task_type, task_data, False, wid, we)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	173
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	174 put_start_time = time.perf_counter()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	175 put(res)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	176 time_in_put += (time.perf_counter() - put_start_time)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	177
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	178 completed += 1
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	179
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	180 # End task... gather stats to send back to the main process.
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	181 elif task_type == TASK_END:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	182 logger.debug("Worker %d got end task, exiting." % wid)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	183 stats.registerTimer('WorkerTaskGet', time=time_in_get)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	184 stats.registerTimer('WorkerResultPut', time=time_in_put)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	185 try:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	186 stats.mergeStats(w.getStats())
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	187 rep = (task_type, task_data, True, wid, (wid, stats))
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	188 except Exception as e:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	189 logger.debug(
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	190 "Error getting report, sending exception to main process:")
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	191 logger.debug(traceback.format_exc())
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	192 we = WorkerExceptionData(wid)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	193 rep = (task_type, task_data, False, wid, (wid, we))
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	194 put(rep)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	195 break
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	196
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	197 # Emergy abort.
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	198 elif task_type == _TASK_ABORT_WORKER:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	199 logger.debug("Worker %d got abort signal." % wid)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	200 break
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	201
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	202 else:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	203 raise Exception("Unknown task type: %s" % task_type)
693 d2a87365b85b bake: Use threads to read/write from/to the main arbitrator process. Ludovic Chabant <ludovic@chabant.com> parents: 691 diff changeset	204
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	205 try:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	206 w.shutdown()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	207 except Exception as e:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	208 logger.error("Worker %s failed to shutdown.")
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	209 logger.exception(e)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	210 raise
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	211
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	212 logger.debug("Worker %d completed %d tasks." % (wid, completed))
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	213
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	214
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	215 class _WorkerParams:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	216 def __init__(self, wid, inqueue, outqueue, worker_class, initargs=(),
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	217 is_profiling=False, is_unit_testing=False):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	218 self.wid = wid
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	219 self.inqueue = inqueue
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	220 self.outqueue = outqueue
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	221 self.worker_class = worker_class
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	222 self.initargs = initargs
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	223 self.is_profiling = is_profiling
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	224 self.is_unit_testing = is_unit_testing
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	225
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	226
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	227 class WorkerPool:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	228 def __init__(self, worker_class, initargs=(), *,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	229 callback=None, error_callback=None,
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	230 worker_count=None, batch_size=None,
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	231 userdata=None):
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	232 self.userdata = userdata
08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	233
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	234 worker_count = worker_count or os.cpu_count() or 1
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	235
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	236 if use_fastqueue:
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	237 self._task_queue = FastQueue()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	238 self._result_queue = FastQueue()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	239 self._quick_put = self._task_queue.put
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	240 self._quick_get = self._result_queue.get
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	241 else:
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	242 self._task_queue = multiprocessing.SimpleQueue()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	243 self._result_queue = multiprocessing.SimpleQueue()
878 313db67cfc35 bake: Use the basic Python queue again. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	244 self._quick_put = self._task_queue.put
313db67cfc35 bake: Use the basic Python queue again. Ludovic Chabant <ludovic@chabant.com> parents: 854 diff changeset	245 self._quick_get = self._result_queue.get
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	246
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	247 self._callback = callback
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	248 self._error_callback = error_callback
462 04abc97dd3b6 bake: Add CLI argument to specify job batch size. Ludovic Chabant <ludovic@chabant.com> parents: 461 diff changeset	249 self._batch_size = batch_size
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	250 self._jobs_left = 0
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	251 self._lock_jobs_left = threading.Lock()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	252 self._lock_workers = threading.Lock()
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	253 self._event = threading.Event()
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	254 self._error_on_join = None
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	255 self._closed = False
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	256
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	257 main_module = sys.modules['__main__']
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	258 is_profiling = os.path.basename(main_module.__file__) in [
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	259 'profile.py', 'cProfile.py']
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	260 is_unit_testing = os.path.basename(main_module.__file__) in [
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	261 'py.test']
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	262
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	263 self._pool = []
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	264 for i in range(worker_count):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	265 worker_params = _WorkerParams(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	266 i, self._task_queue, self._result_queue,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	267 worker_class, initargs,
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	268 is_profiling=is_profiling,
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	269 is_unit_testing=is_unit_testing)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	270 w = multiprocessing.Process(target=worker_func,
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	271 args=(worker_params,))
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	272 w.name = w.name.replace('Process', 'PoolWorker')
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	273 w.daemon = True
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	274 w.start()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	275 self._pool.append(w)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	276
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	277 self._result_handler = threading.Thread(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	278 target=WorkerPool._handleResults,
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	279 args=(self,))
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	280 self._result_handler.daemon = True
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	281 self._result_handler.start()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	282
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	283 def queueJobs(self, jobs):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	284 if self._closed:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	285 if self._error_on_join:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	286 raise self._error_on_join
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	287 raise Exception("This worker pool has been closed.")
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	288
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	289 jobs = list(jobs)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	290 new_job_count = len(jobs)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	291 if new_job_count > 0:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	292 with self._lock_jobs_left:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	293 self._jobs_left += new_job_count
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	294
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	295 self._event.clear()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	296 for job in jobs:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	297 self._quick_put((TASK_JOB, job))
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	298
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	299 def wait(self, timeout=None):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	300 if self._closed:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	301 raise Exception("This worker pool has been closed.")
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	302
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	303 ret = self._event.wait(timeout)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	304 if self._error_on_join:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	305 raise self._error_on_join
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	306 return ret
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	307
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	308 def close(self):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	309 if self._closed:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	310 raise Exception("This worker pool has been closed.")
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	311 if self._jobs_left > 0 or not self._event.is_set():
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	312 raise Exception("A previous job queue has not finished yet.")
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	313
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	314 logger.debug("Closing worker pool...")
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	315 live_workers = list(filter(lambda w: w is not None, self._pool))
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	316 handler = _ReportHandler(len(live_workers))
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	317 self._callback = handler._handle
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	318 self._error_callback = handler._handleError
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	319 for w in live_workers:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	320 self._quick_put((TASK_END, None))
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	321 for w in live_workers:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	322 w.join()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	323
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	324 logger.debug("Waiting for reports...")
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	325 if not handler.wait(2):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	326 missing = handler.reports.index(None)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	327 logger.warning(
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	328 "Didn't receive all worker reports before timeout. "
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	329 "Missing report from worker %d." % missing)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	330
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	331 logger.debug("Exiting result handler thread...")
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	332 self._result_queue.put(None)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	333 self._result_handler.join()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	334 self._closed = True
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	335
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	336 return handler.reports
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	337
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	338 def _onResultHandlerCriticalError(self, wid):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	339 logger.error("Result handler received a critical error from "
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	340 "worker %d." % wid)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	341 with self._lock_workers:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	342 self._pool[wid] = None
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	343 if all(map(lambda w: w is None, self._pool)):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	344 logger.error("All workers have died!")
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	345 self._closed = True
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	346 self._error_on_join = Exception("All workers have died!")
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	347 self._event.set()
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	348 return False
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	349
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	350 return True
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	351
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	352 def _onTaskDone(self):
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	353 with self._lock_jobs_left:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	354 left = self._jobs_left - 1
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	355 self._jobs_left = left
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	356
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	357 if left == 0:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	358 self._event.set()
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	359
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	360 @staticmethod
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	361 def _handleResults(pool):
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	362 userdata = pool.userdata
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	363 while True:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	364 try:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	365 res = pool._quick_get()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	366 except (EOFError, OSError):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	367 logger.debug("Result handler thread encountered connection "
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	368 "problem, exiting.")
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	369 return
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	370
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	371 if res is None:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	372 logger.debug("Result handler exiting.")
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	373 return
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	374
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	375 task_type, task_data, success, wid, data = res
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	376 try:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	377 if success:
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	378 if pool._callback:
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	379 pool._callback(task_data, data, userdata)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	380 else:
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	381 if task_type == _CRITICAL_WORKER_ERROR:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	382 logger.error(data)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	383 do_continue = pool._onResultHandlerCriticalError(wid)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	384 if not do_continue:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	385 logger.debug("Aborting result handling thread.")
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	386 return
451 838f3964f400 bake: Optimize the bake by not using custom classes for passing info. Ludovic Chabant <ludovic@chabant.com> parents: 447 diff changeset	387 else:
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	388 if pool._error_callback:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	389 pool._error_callback(task_data, data, userdata)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	390 else:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	391 logger.error(
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	392 "Worker %d failed to process a job:" % wid)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	393 logger.error(data)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	394 except Exception as ex:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	395 logger.exception(ex)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	396
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	397 if task_type == TASK_JOB:
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	398 pool._onTaskDone()
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	399
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	400
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	401 class _ReportHandler:
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	402 def __init__(self, worker_count):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	403 self.reports = [None] * worker_count
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	404 self._count = worker_count
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	405 self._received = 0
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	406 self._event = threading.Event()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	407
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	408 def wait(self, timeout=None):
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	409 return self._event.wait(timeout)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	410
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	411 def _handle(self, job, res, _):
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	412 wid, data = res
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	413 if wid < 0 or wid > self._count:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	414 logger.error("Ignoring report from unknown worker %d." % wid)
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	415 return
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	416
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	417 self._received += 1
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	418 self.reports[wid] = data
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	419
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	420 if self._received == self._count:
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	421 self._event.set()
aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	422
854 08e02c2a2a1a core: Keep refactoring, this time to prepare for generator sources. Ludovic Chabant <ludovic@chabant.com> parents: 852 diff changeset	423 def _handleError(self, job, res, _):
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	424 logger.error("Worker %d failed to send its report." % res.wid)
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	425 logger.error(res)
447 aefe70229fdd bake: Commonize worker pool code between html and asset baking. Ludovic Chabant <ludovic@chabant.com> parents: diff changeset	426
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	427
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	428 class FastQueue:
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	429 def __init__(self):
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	430 self._reader, self._writer = multiprocessing.Pipe(duplex=False)
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	431 self._rlock = multiprocessing.Lock()
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	432 self._wlock = multiprocessing.Lock()
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	433 self._initBuffers()
9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	434
9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	435 def _initBuffers(self):
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	436 self._rbuf = io.BytesIO()
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	437 self._rbuf.truncate(256)
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	438 self._wbuf = io.BytesIO()
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	439 self._wbuf.truncate(256)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	440
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	441 def __getstate__(self):
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	442 return (self._reader, self._writer, self._rlock, self._wlock)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	443
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	444 def __setstate__(self, state):
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	445 (self._reader, self._writer, self._rlock, self._wlock) = state
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	446 self._initBuffers()
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	447
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	448 def get(self):
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	449 with self._rlock:
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	450 try:
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	451 with self._rbuf.getbuffer() as b:
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	452 bufsize = self._reader.recv_bytes_into(b)
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	453 except multiprocessing.BufferTooShort as e:
697 9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	454 bufsize = len(e.args[0])
9e5393fcfab2 bake: Re-enable faster serialization between processes. Ludovic Chabant <ludovic@chabant.com> parents: 693 diff changeset	455 self._rbuf.truncate(bufsize * 2)
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	456 self._rbuf.seek(0)
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	457 self._rbuf.write(e.args[0])
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	458
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	459 self._rbuf.seek(0)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	460 return _unpickle(self._rbuf, bufsize)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	461
b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	462 def put(self, obj):
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	463 self._wbuf.seek(0)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	464 _pickle(obj, self._wbuf)
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	465 size = self._wbuf.tell()
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	466
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	467 self._wbuf.seek(0)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	468 with self._wlock:
691 9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	469 with self._wbuf.getbuffer() as b:
9ae9390192da bake: Use standard pickle and queue for now to fix some small issues. Ludovic Chabant <ludovic@chabant.com> parents: 687 diff changeset	470 self._writer.send_bytes(b, 0, size)
461 b015e38d4ee1 internal: Handle data serialization more under the hood. Ludovic Chabant <ludovic@chabant.com> parents: 460 diff changeset	471
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	472
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	473 if use_fastpickle:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	474 from piecrust import fastpickle
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	475
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	476 def _pickle_fast(obj, buf):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	477 fastpickle.pickle_intob(obj, buf)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	478
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	479 def _unpickle_fast(buf, bufsize):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	480 return fastpickle.unpickle_fromb(buf, bufsize)
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	481
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	482 _pickle = _pickle_fast
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	483 _unpickle = _unpickle_fast
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	484
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	485 elif use_msgpack:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	486 import msgpack
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	487
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	488 def _pickle_msgpack(obj, buf):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	489 msgpack.pack(obj, buf)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	490
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	491 def _unpickle_msgpack(buf, bufsize):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	492 return msgpack.unpack(buf)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	493
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	494 _pickle = _pickle_msgpack
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	495 _unpickle = _unpickle_msgpack
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	496
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	497 elif use_marshall:
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	498 import marshal
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	499
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	500 def _pickle_marshal(obj, buf):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	501 marshal.dump(obj, buf)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	502
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	503 def _unpickle_marshal(buf, bufsize):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	504 return marshal.load(buf)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	505
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	506 _pickle = _pickle_marshal
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	507 _unpickle = _unpickle_marshal
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	508
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	509 else:
971 5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	510 import pickle
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	511
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	512 def _pickle_default(obj, buf):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	513 pickle.dump(obj, buf, pickle.HIGHEST_PROTOCOL)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	514
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	515 def _unpickle_default(buf, bufsize):
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	516 return pickle.load(buf)
5485a11591ec internal: Worker pool improvements Ludovic Chabant <ludovic@chabant.com> parents: 911 diff changeset	517
852 4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	518 _pickle = _pickle_default
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	519 _unpickle = _unpickle_default
4850f8c21b6e core: Start of the big refactor for PieCrust 3.0. Ludovic Chabant <ludovic@chabant.com> parents: 702 diff changeset	520

Mercurial > piecrust2

annotate piecrust/workerpool.py @ 978:7e51d14097cb