tpcp.validate.Scorer#

class tpcp.validate.Scorer(score_func: ~typing.Callable[[~tpcp._pipeline.PipelineT, ~tpcp._dataset.DatasetT], ~typing.Union[~tpcp.validate._scorer.T, Aggregator[Any], ~typing.Dict[str, ~typing.Union[~tpcp.validate._scorer.T, Aggregator[Any]]], ~typing.Dict[str, ~typing.Union[~tpcp.validate._scorer.T, Aggregator[Any], ~typing.Dict[str, ~typing.Union[~tpcp.validate._scorer.T, Aggregator[Any]]]]]]], *, default_aggregator: ~typing.Type[~tpcp.validate._scorer.Aggregator[~tpcp.validate._scorer.T]] = <class 'tpcp.validate._scorer.MeanAggregator'>, single_score_callback: ~typing.Optional[~tpcp.validate._scorer.ScoreCallback[~tpcp._pipeline.PipelineT, ~tpcp._dataset.DatasetT, ~tpcp.validate._scorer.T]] = None, **kwargs: ~typing.Any)[source]#

A scorer to score multiple data points of a dataset and average the results.

Parameters
score_func

The callable that is used to score each data point

single_score_callback

Callback function that is called after each datapoint that is scored. It should have the following call signature:

>>> def callback(
...     *,
...     step: int,
...     scores: Tuple[_SCORE_TYPE, ...],
...     scorer: "Scorer",
...     pipeline: Pipeline,
...     dataset: Dataset,
...     **_
... ) -> None:
...     ...

All parameters will be passed as keyword arguments. This means, if your callback only needs a subset of the defined parameters, you can ignore them by using unused kwargs:

>>> def callback(*, step: int, pipeline: Pipeline, **_):
...     ...
kwargs

Additional arguments that might be used by the scorer. These are ignored for the base scorer.

Methods

__call__(pipeline, dataset)

Score the pipeline with the provided data.
__init__(score_func: ~typing.Callable[[~tpcp._pipeline.PipelineT, ~tpcp._dataset.DatasetT], ~typing.Union[~tpcp.validate._scorer.T, Aggregator[Any], ~typing.Dict[str, ~typing.Union[~tpcp.validate._scorer.T, Aggregator[Any]]], ~typing.Dict[str, ~typing.Union[~tpcp.validate._scorer.T, Aggregator[Any], ~typing.Dict[str, ~typing.Union[~tpcp.validate._scorer.T, Aggregator[Any]]]]]]], *, default_aggregator: ~typing.Type[~tpcp.validate._scorer.Aggregator[~tpcp.validate._scorer.T]] = <class 'tpcp.validate._scorer.MeanAggregator'>, single_score_callback: ~typing.Optional[~tpcp.validate._scorer.ScoreCallback[~tpcp._pipeline.PipelineT, ~tpcp._dataset.DatasetT, ~tpcp.validate._scorer.T]] = None, **kwargs: ~typing.Any)[source]#

Examples using tpcp.validate.Scorer#

Grid Search optimal Algorithm Parameter

Grid Search optimal Algorithm Parameter

Grid Search optimal Algorithm Parameter
Custom Optuna Optimizer

Custom Optuna Optimizer

Custom Optuna Optimizer
Custom Scorer

Custom Scorer

Custom Scorer