Scorer

class tpcp.validate.Scorer(score_func: ~typing.Callable[[~tpcp._pipeline.PipelineT, ~tpcp._dataset.DatasetT], ~tpcp.validate._scorer.T | ~tpcp.validate._scorer.Aggregator[~typing.Any] | dict[str, ~tpcp.validate._scorer.T | ~tpcp.validate._scorer.Aggregator[~typing.Any]] | dict[str, ~tpcp.validate._scorer.T | ~tpcp.validate._scorer.Aggregator[~typing.Any] | dict[str, ~tpcp.validate._scorer.T | ~tpcp.validate._scorer.Aggregator[~typing.Any]]]], *, default_aggregator: type[~tpcp.validate._scorer.Aggregator[~tpcp.validate._scorer.T]] = <class 'tpcp.validate._scorer.MeanAggregator'>, single_score_callback: ~tpcp.validate._scorer.ScoreCallback[~tpcp._pipeline.PipelineT, ~tpcp._dataset.DatasetT, ~tpcp.validate._scorer.T] | None = None, n_jobs: int | None = None, verbose: int = 0, pre_dispatch: str | int = '2*n_jobs', progress_bar: bool = True)

A scorer to score multiple data points of a dataset and average the results.

Parameters:

score_func

The callable that is used to score each data point

single_score_callback

Callback function that is called after each datapoint that is scored. It should have the following call signature:

>>> def callback(
...     *, step: int, scores: Tuple[_SCORE_TYPE, ...], scorer: "Scorer", pipeline: Pipeline, dataset: Dataset, **_
... ) -> None:
...     ...

All parameters will be passed as keyword arguments. This means, if your callback only needs a subset of the defined parameters, you can ignore them by using unused kwargs:

>>> def callback(*, step: int, pipeline: Pipeline, **_):
...     ...

n_jobs

The number of parallel jobs to run. Each job will run on a single data point. Note, that the single_score_callback will still be called in the main thread, after a job is finished. However, it could be that multiple jobs are finished before the callback is called. The callback is still gurateed to be called in the order of the data points. If None, no parallelization is used.

verbose

Controls the verbosity of the parallelization. See joblib.Parallel for more details.

pre_dispatch

Controls the number of jobs that get dispatched during parallelization. See joblib.Parallel for more details.

progress_bar

True/False to enable/disable a tqdm progress bar.

Methods

__call__(pipeline, dataset)	Score the pipeline with the provided data.
clone()	Create a new instance of the class with all parameters copied over.
get_params([deep])	Get parameters for this algorithm.
set_params(**params)	Set the parameters of this Algorithm.

__init__(score_func: ~typing.Callable[[~tpcp._pipeline.PipelineT, ~tpcp._dataset.DatasetT], ~tpcp.validate._scorer.T | ~tpcp.validate._scorer.Aggregator[~typing.Any] | dict[str, ~tpcp.validate._scorer.T | ~tpcp.validate._scorer.Aggregator[~typing.Any]] | dict[str, ~tpcp.validate._scorer.T | ~tpcp.validate._scorer.Aggregator[~typing.Any] | dict[str, ~tpcp.validate._scorer.T | ~tpcp.validate._scorer.Aggregator[~typing.Any]]]], *, default_aggregator: type[~tpcp.validate._scorer.Aggregator[~tpcp.validate._scorer.T]] = <class 'tpcp.validate._scorer.MeanAggregator'>, single_score_callback: ~tpcp.validate._scorer.ScoreCallback[~tpcp._pipeline.PipelineT, ~tpcp._dataset.DatasetT, ~tpcp.validate._scorer.T] | None = None, n_jobs: int | None = None, verbose: int = 0, pre_dispatch: str | int = '2*n_jobs', progress_bar: bool = True) → None

clone() → Self

Create a new instance of the class with all parameters copied over.

This will create a new instance of the class itself and all nested objects

get_params(deep: bool = True) → dict[str, Any]

Get parameters for this algorithm.

Parameters:

deep

Only relevant if object contains nested algorithm objects. If this is the case and deep is True, the params of these nested objects are included in the output using a prefix like nested_object_name__ (Note the two “_” at the end)

Returns:

params

Parameter names mapped to their values.

set_params(**params: Any) → Self

Set the parameters of this Algorithm.

To set parameters of nested objects use nested_object_name__para_name=.

Examples using tpcp.validate.Scorer

Custom Optuna Optimizer

Custom Optuna Optimizer

Custom Scorer

Custom Scorer