Parallel

class tpcp.parallel.Parallel(n_jobs=default(None), backend=default(None), return_as='list', verbose=default(0), timeout=None, pre_dispatch='2 * n_jobs', batch_size='auto', temp_folder=default(None), max_nbytes=default('1M'), mmap_mode=default('r'), prefer=default(None), require=default(None), **backend_kwargs)

Run joblib tasks and recover side-channel data from TPCP worker contexts.

This is a drop-in joblib.Parallel subclass. When paired with delayed, side-channel data produced by a successfully completed worker task is merged back into parent-process state. This includes records produced by restored warning/error contexts. Generator results are merged lazily as they are consumed.

Side-channel data from a task that raises is not recovered; joblib propagates the original exception unchanged. The exception still retains context added inside the worker. Plain joblib.Parallel remains supported with delayed, but does not recover worker side-channel data.

Methods

__call__(iterable)	Dispatch delayed tasks and merge side-channel data in the parent.
dispatch_next()	Dispatch more data for parallel processing
dispatch_one_batch(iterator)	Prefetch the tasks for the next batch and dispatch them.
format(obj[, indent])	Return the formatted representation of the object.
print_progress()	Display the process of the parallel execution only a fraction of time, controlled by self.verbose.

debug
info
warn

__init__(n_jobs=default(None), backend=default(None), return_as='list', verbose=default(0), timeout=None, pre_dispatch='2 * n_jobs', batch_size='auto', temp_folder=default(None), max_nbytes=default('1M'), mmap_mode=default('r'), prefer=default(None), require=default(None), **backend_kwargs)

Parameters:

depth: int, optional

The depth of objects printed.

name: str, optional

The namespace to log to. If None, defaults to joblib.

dispatch_next()

Dispatch more data for parallel processing

This method is meant to be called concurrently by the multiprocessing callback. We rely on the thread-safety of dispatch_one_batch to protect against concurrent consumption of the unprotected iterator.

dispatch_one_batch(iterator)

Prefetch the tasks for the next batch and dispatch them.

The effective size of the batch is computed here. If there are no more jobs to dispatch, return False, else return True.

The iterator consumption and dispatching is protected by the same lock so calling this function should be thread safe.

format(obj, indent=0)

Return the formatted representation of the object.

print_progress()

Display the process of the parallel execution only a fraction of time, controlled by self.verbose.

Examples using tpcp.parallel.Parallel

Caching

Caching

Warning and Error Context

Warning and Error Context