tpcp.validate.Scorer

class tpcp.validate.Scorer(score_func: Callable[[tpcp._pipeline.PipelineT, tpcp._dataset.DatasetT], tpcp.validate._scorer.ScoreTypeT], *, single_score_callback: Optional[tpcp.validate._scorer.ScoreCallback[tpcp._pipeline.PipelineT, tpcp._dataset.DatasetT, tpcp.validate._scorer.ScoreTypeT]] = None, **kwargs: Any)

A scorer to score multiple data points of a dataset and average the results.

Parameters

score_func

The callable that is used to score each data point

single_score_callback

Callback function that is called after each datapoint that is scored. It should have the following call signature:

>>> def callback(
...     *,
...     step: int,
...     scores: Tuple[_SCORE_TYPE, ...],
...     scorer: "Scorer",
...     pipeline: Pipeline,
...     dataset: Dataset,
...     error_score: _ERROR_SCORE_TYPE,
...     **_
... ) -> None:
...     ...

All parameters will be passed as keyword arguments. This means, if your callback only needs a subset of the defined parameters, you can ignore them by using unused kwargs:

>>> def callback(*, step: int, pipeline: Pipeline, **_):
...     ...

kwargs

Additional arguments that might be used by the scorer. These are ignored for the base scorer.

Methods

__call__(pipeline, dataset, error_score)	Score the pipeline with the provided data.
aggregate(scores)	Aggregate the scores of each data point.

__init__(score_func: Callable[[tpcp._pipeline.PipelineT, tpcp._dataset.DatasetT], tpcp.validate._scorer.ScoreTypeT], *, single_score_callback: Optional[tpcp.validate._scorer.ScoreCallback[tpcp._pipeline.PipelineT, tpcp._dataset.DatasetT, tpcp.validate._scorer.ScoreTypeT]] = None, **kwargs: Any)

aggregate(scores: Sequence[float]) → float

Aggregate the scores of each data point.

Examples using tpcp.validate.Scorer

Grid Search optimal Algorithm Parameter

Custom Optuna Optimizer