tpcp.validate.Scorer

class tpcp.validate.Scorer(score_func: ~typing.Callable[[~tpcp._pipeline.PipelineT, ~tpcp._dataset.DatasetT], ~tpcp.validate._scorer.T | Aggregator[Any] | ~typing.Dict[str, ~tpcp.validate._scorer.T | Aggregator[Any]] | ~typing.Dict[str, ~tpcp.validate._scorer.T | Aggregator[Any] | ~typing.Dict[str, ~tpcp.validate._scorer.T | Aggregator[Any]]]], *, default_aggregator: ~typing.Type[~tpcp.validate._scorer.Aggregator[~tpcp.validate._scorer.T]] = <class 'tpcp.validate._scorer.MeanAggregator'>, single_score_callback: ~tpcp.validate._scorer.ScoreCallback[~tpcp._pipeline.PipelineT, ~tpcp._dataset.DatasetT, ~tpcp.validate._scorer.T] | None = None, **kwargs: ~typing.Any)

A scorer to score multiple data points of a dataset and average the results.

Parameters:

score_func

The callable that is used to score each data point

single_score_callback

Callback function that is called after each datapoint that is scored. It should have the following call signature:

>>> def callback(
...     *,
...     step: int,
...     scores: Tuple[_SCORE_TYPE, ...],
...     scorer: "Scorer",
...     pipeline: Pipeline,
...     dataset: Dataset,
...     **_
... ) -> None:
...     ...

All parameters will be passed as keyword arguments. This means, if your callback only needs a subset of the defined parameters, you can ignore them by using unused kwargs:

>>> def callback(*, step: int, pipeline: Pipeline, **_):
...     ...

kwargs

Additional arguments that might be used by the scorer. These are ignored for the base scorer.

Methods

__call__(pipeline, dataset)

Score the pipeline with the provided data.

__init__(score_func: ~typing.Callable[[~tpcp._pipeline.PipelineT, ~tpcp._dataset.DatasetT], ~tpcp.validate._scorer.T | Aggregator[Any] | ~typing.Dict[str, ~tpcp.validate._scorer.T | Aggregator[Any]] | ~typing.Dict[str, ~tpcp.validate._scorer.T | Aggregator[Any] | ~typing.Dict[str, ~tpcp.validate._scorer.T | Aggregator[Any]]]], *, default_aggregator: ~typing.Type[~tpcp.validate._scorer.Aggregator[~tpcp.validate._scorer.T]] = <class 'tpcp.validate._scorer.MeanAggregator'>, single_score_callback: ~tpcp.validate._scorer.ScoreCallback[~tpcp._pipeline.PipelineT, ~tpcp._dataset.DatasetT, ~tpcp.validate._scorer.T] | None = None, **kwargs: ~typing.Any)

Examples using tpcp.validate.Scorer

Custom Optuna Optimizer

Custom Optuna Optimizer

Custom Scorer

Custom Scorer