cross_validate#

tpcp.validate.cross_validate(optimizable: BaseOptimize, dataset: Dataset, *, scoring: Callable | None = None, cv: DatasetSplitter | int | BaseCrossValidator | Iterator | None = None, n_jobs: int | None = None, verbose: int = 0, optimize_params: dict[str, Any] | None = None, pre_dispatch: str | int = '2*n_jobs', return_train_score: bool = False, return_optimizer: bool = False, progress_bar: bool = True)[source]#

Evaluate a pipeline on a dataset using cross validation.

This function follows as much as possible the interface of cross_validate. If the tpcp documentation is missing some information, the respective documentation of sklearn might be helpful.

Parameters:

optimizable

A optimizable class instance like GridSearch/GridSearchCV or a Pipeline wrapped in an Optimize object (OptimizablePipeline).

dataset

A Dataset containing all information.

scoring

A callable that can score a single data point given a pipeline. This function should return either a single score or a dictionary of scores. If scoring is None the default score method of the optimizable is used instead.

cv

The cross-validation strategy to use. For simple use-cases the same input as for the sklearn cross-validation function are supported. For further inputs check the sklearn documentation.

For more complex usecases like grouping or stratification, the TpcpSplitter can be used.

n_jobs

Number of jobs to run in parallel. One job is created per CV fold. The default (None) means 1 job at the time, hence, no parallel computing.

verbose

The verbosity level (larger number -> higher verbosity). At the moment this only effects Parallel.

optimize_params

Additional parameter that are forwarded to the optimize method.

pre_dispatch

The number of jobs that should be pre dispatched. For an explanation see the documentation of Parallel.

return_train_score

If True the performance on the train score is returned in addition to the test score performance. Note, that this increases the runtime. If True, the fields train_data_labels, train_score, and train_score_single are available in the results.

return_optimizer

If the optimized instance of the input optimizable should be returned. If True, the field optimizer is available in the results.

progress_bar

True/False to enable/disable a tqdm progress bar.

Returns:

result_dict

Dictionary with results. Each element is either a list or array of length n_folds. The dictionary can be directly passed into the pandas DataFrame constructor for a better representation.

The following fields are in the results:

test__score / test__{scorer-name}: The aggregated value of a score over all data-points. If a single score is used for scoring, then the generic name “score” is used. Otherwise, multiple columns with the name of the respective scorer exist.
test__single__score / test__single__{scorer-name}: The individual scores per datapoint per fold. This is a list of values with the len(train_set).
test__data_labels: A list of data labels of the train set in the order the single score values are provided. These can be used to associate the single_score values with a certain data-point.
train__score / train__{scorer-name}: Results for train set of each fold.
train__single__score / train__single__{scorer-name}: Results for individual data points in the train set of each fold
train__data_labels: The data labels for the train set.
optimize_time: Time required to optimize the pipeline in each fold.
score_time: Cumulative score time to score all data points in the test set.
optimizer: The optimized instances per fold. One instance per fold is returned. The optimized version of the pipeline can be obtained via the optimized_pipeline_ attribute on the instance.