tpcp.parallel: Context-aware parallel execution

For practical caveats around joblib-based multiprocessing, worker state, serialization, and import costs, see Multiprocessing Caveats.

Helpers for restoring TPCP runtime context in joblib workers.

The worker-state helpers are required as long as joblib/joblib#1071 is not resolved. The delayed wrapper restores registered global state, Python warning filters, and active warning/error contexts for each worker task. Parallel additionally recovers side-channel data from successful tasks without changing their return values. Applications can add their own round-trip state with register_parallel_side_channel.

The provided workarounds are similar to the ones done in scikit-learn (scikit-learn/scikit-learn#25363).

Note, that these fixes are not necessarily compatible with each other. This means you can not forward custom callbacks through scikit-learn Parallel calls. The same way, by default tpcp will not forward sklearn global configs through tpcp Parallel calls. However, you can likely configure the callbacks in tpcp to make that work.

Functions

delayed(func)	Wrap a function to be used in a parallel context.
register_global_parallel_callback(callback)	Register a callback to transfer information to parallel workers.
register_parallel_side_channel(capture, ...)	Register state that can make a round trip through a parallel worker.
remove_global_parallel_callback(name)	Remove a registered callback.
remove_parallel_side_channel(name)	Remove a side channel registered with register_parallel_side_channel.

Classes

Parallel([n_jobs, backend, return_as, ...])

Run joblib tasks and recover side-channel data from TPCP worker contexts.