Note

Go to the end to download the full example code

Tensorflow/Keras#

Note

This example requires the tensorflow package to be installed.

Theoretically, tpcp is framework agnostic and can be used with any framework. However, due to the way some frameworks handle their objects, some special handling internally is required. Hence, this example does not only serve as example on how to use tensorflow with tpcp, but also as a test case for these special cases.

When using tpcp with any machine learning framework, you either want to use a pretrained model with a normal pipeline or a train your own model as part of an Optimizable Pipeline. Here we show the second case, as it is more complex, and you are likely able to figure out the first case yourself.

This means, we are planning to perform the following steps:

  1. Create a pipeline that creates and trains a model.

  2. Allow the modification of model hyperparameters.

  3. Run a simple cross-validation to demonstrate the functionality.

This example reimplements the basic MNIST example from the [tensorflow documentation](https://www.tensorflow.org/tutorials/keras/classification).

Some Notes#

In this example we show how to implement a Pipeline that uses tensorflow. You could implement an Algorithm in a similar way. This would actually be easier, as no specific handling of the input data would be required. For a pipeline, we need to create a custom Dataset class, as this is the expected input for a pipeline.

The Dataset#

We are using the normal fashion MNIST dataset for this example It consists of 60.000 images of 28x28 pixels, each with a label. We will ignore the typical train-test split, as we want to do our own cross-validation.

In addition, we will simulate an additional “index level”. In this (and most typical deep learning datasets), each datapoint is one vector for which we can make one prediction. In tpcp, we usually deal with datasets, where you might have multiple pieces of information for each datapoint. For example, one datapoint could be a patient, for which we have an entire time series of measurements. We will simulate this here, by creating the index of our dataset as 1000 groups each containing 60 images.

Other than that, the dataset is pretty standard. Besides the create_index method, we only need to implement the input_as_array and labels_as_array methods that allow us to easily access the data once we selected a single group.

from functools import lru_cache

import numpy as np
import pandas as pd
import tensorflow as tf

from tpcp import Dataset

tf.keras.utils.set_random_seed(812)
tf.config.experimental.enable_op_determinism()


@lru_cache(maxsize=1)
def get_fashion_mnist_data():
    # Note: We throw train and test sets together, as we don't care about the official split here.
    #       We will create our own split later.
    (train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.fashion_mnist.load_data()
    return np.array(list(train_images) + list(test_images)), list(train_labels) + list(test_labels)


class FashionMNIST(Dataset):
    def input_as_array(self) -> np.ndarray:
        self.assert_is_single(None, "input_as_array")
        group_id = int(self.group_label.group_id)
        images, _ = get_fashion_mnist_data()
        return images[group_id * 60 : (group_id + 1) * 60].reshape((60, 28, 28)) / 255

    def labels_as_array(self) -> np.ndarray:
        self.assert_is_single(None, "labels_as_array")
        group_id = int(self.group_label.group_id)
        _, labels = get_fashion_mnist_data()
        return np.array(labels[group_id * 60 : (group_id + 1) * 60])

    def create_index(self) -> pd.DataFrame:
        # There are 60.000 images in total.
        # We simulate 1000 groups of 60 images each.
        return pd.DataFrame({"group_id": list(range(1000))})

We can see our Dataset works as expected:

dataset = FashionMNIST()
dataset[0].input_as_array().shape

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz

 8192/29515 [=======>......................] - ETA: 0s
29515/29515 [==============================] - 0s 1us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz

    8192/26421880 [..............................] - ETA: 0s
  131072/26421880 [..............................] - ETA: 10s
  999424/26421880 [>.............................] - ETA: 2s 
 5242880/26421880 [====>.........................] - ETA: 0s
10059776/26421880 [==========>...................] - ETA: 0s
14417920/26421880 [===============>..............] - ETA: 0s
19234816/26421880 [====================>.........] - ETA: 0s
23658496/26421880 [=========================>....] - ETA: 0s
26421880/26421880 [==============================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz

5148/5148 [==============================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz

   8192/4422102 [..............................] - ETA: 0s
 139264/4422102 [..............................] - ETA: 1s
 974848/4422102 [=====>........................] - ETA: 0s
2138112/4422102 [=============>................] - ETA: 0s
4422102/4422102 [==============================] - 0s 0us/step

(60, 28, 28)

dataset[0].labels_as_array().shape

(60,)

The Pipeline#

We will create a pipeline that uses a simple neural network to classify the images. In tpcp, all “things” that should be optimized need to be parameters. This means our model itself needs to be a parameter of the pipeline. However, as we don’t have the model yet, as its creation depends on other hyperparameters, we add it as an optional parameter initialized with None. Further, we prefix the parameter name with an underscore, to signify, that this is not a parameter that should be modified manually by the user. This is just convention, and it is up to you to decide how you want to name your parameters.

We further introduce a hyperparameter n_dense_layer_nodes to show how we can influence the model creation.

The optimize method#

To make our pipeline optimizable, it needs to inherit from OptimizablePipeline. Further we need to mark at least one of the parameters as OptiPara using the type annotation. We do this for our _model parameter.

Finally, we need to implement the self_optimize method. This method will get the entire training dataset as input and should update the _model parameter with the trained model. Hence, we first extract the relevant data (remember, each datapoint is 60 images), by concatinating all images over all groups in the dataset. Then we create the Keras model based on the hyperparameters. Finally, we train the model and update the _model parameter.

Here we chose to wrap the method with make_optimize_safe. This decorator will perform some runtime checks to ensure that the method is implemented correctly.

The run method#

The run method expects that the _model parameter is already set (i.e. the pipeline was already optimized). It gets a single datapoint as input (remember, a datapoint is a single group of 60 images). We then extract the data from the datapoint and let the model make a prediction. We store the prediction on our output attribute predictions_. The trailing underscore is a convention to signify, that this is an “result” attribute.

import warnings
from typing import Optional

from typing_extensions import Self

from tpcp import OptimizablePipeline, OptiPara, make_action_safe, make_optimize_safe


class KerasPipeline(OptimizablePipeline):
    n_dense_layer_nodes: int
    n_train_epochs: int
    _model: OptiPara[Optional[tf.keras.Sequential]]

    predictions_: np.ndarray

    def __init__(self, n_dense_layer_nodes=128, n_train_epochs=5, _model: Optional[tf.keras.Sequential] = None):
        self.n_dense_layer_nodes = n_dense_layer_nodes
        self.n_train_epochs = n_train_epochs
        self._model = _model

    @property
    def predicted_labels_(self):
        return np.argmax(self.predictions_, axis=1)

    @make_optimize_safe
    def self_optimize(self, dataset, **_) -> Self:
        data = np.vstack([d.input_as_array() for d in dataset])
        labels = np.hstack([d.labels_as_array() for d in dataset])

        print(data.shape)
        if self._model is not None:
            warnings.warn("Overwriting existing model!")

        self._model = tf.keras.Sequential(
            [
                tf.keras.layers.Flatten(input_shape=(28, 28)),
                tf.keras.layers.Dense(self.n_dense_layer_nodes, activation="relu"),
                tf.keras.layers.Dense(10),
            ]
        )

        self._model.compile(
            optimizer="adam",
            loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
            metrics=["accuracy"],
        )

        self._model.fit(data, labels, epochs=self.n_train_epochs)

        return self

    @make_action_safe
    def run(self, datapoint) -> Self:
        if self._model is None:
            raise RuntimeError("Model not trained yet!")
        data = datapoint.input_as_array()

        self.predictions_ = self._model.predict(data)
        return self

Testing the pipeline#

We can now test our pipeline. We will run the optimization using a couple of datapoints (to keep everything fast) and then use run to get the predictions for a single unseen datapoint.

pipeline = KerasPipeline().self_optimize(FashionMNIST()[:10])
p1 = pipeline.run(FashionMNIST()[11])
print(p1.predicted_labels_)
print(FashionMNIST()[11].labels_as_array())

(600, 28, 28)
Epoch 1/5

 1/19 [>.............................] - ETA: 11s - loss: 2.3593 - accuracy: 0.1250
19/19 [==============================] - 1s 3ms/step - loss: 1.5112 - accuracy: 0.4933
Epoch 2/5

 1/19 [>.............................] - ETA: 0s - loss: 0.9220 - accuracy: 0.7188
19/19 [==============================] - 0s 2ms/step - loss: 0.8705 - accuracy: 0.7083
Epoch 3/5

 1/19 [>.............................] - ETA: 0s - loss: 1.0179 - accuracy: 0.6875
19/19 [==============================] - 0s 2ms/step - loss: 0.6799 - accuracy: 0.7850
Epoch 4/5

 1/19 [>.............................] - ETA: 0s - loss: 0.4521 - accuracy: 0.8750
19/19 [==============================] - 0s 2ms/step - loss: 0.5962 - accuracy: 0.8133
Epoch 5/5

 1/19 [>.............................] - ETA: 0s - loss: 0.4230 - accuracy: 0.8438
19/19 [==============================] - 0s 2ms/step - loss: 0.5158 - accuracy: 0.8400

1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step
[8 8 0 9 6 0 7 3 7 9 3 8 6 3 7 8 1 4 0 7 9 8 5 5 2 1 3 3 1 9 7 5 9 9 7 8 2
 7 2 7 2 6 7 1 1 7 5 4 8 3 5 9 0 7 3 0 0 9 1 9]
[8 8 0 9 2 0 7 3 7 9 3 8 4 3 7 8 1 4 0 7 9 8 5 5 2 1 3 4 6 7 7 5 9 9 7 8 2
 7 4 7 0 3 5 1 1 5 5 2 8 3 5 9 0 7 3 0 0 7 1 9]

We can see that even with just 5 epochs, the model already performs quite well. To quantify we can calculate the accuracy for this datapoint:

from sklearn.metrics import accuracy_score

accuracy_score(p1.predicted_labels_, FashionMNIST()[11].labels_as_array())

0.8

Cross Validation#

If we want to run a cross validation, we need to formalize the scoring into a function. We will calculate two types of accuracy: First, the accuracy per group and second, the accuracy over all images across all groups. For more information about how this works, check the Custom Scorer example.

from collections.abc import Sequence

from tpcp.validate import Aggregator


class SingleValueAccuracy(Aggregator[np.ndarray]):
    RETURN_RAW_SCORES = False

    @classmethod
    def aggregate(cls, /, values: Sequence[tuple[np.ndarray, np.ndarray]], **_) -> dict[str, float]:
        return {"accuracy": accuracy_score(np.hstack([v[0] for v in values]), np.hstack([v[1] for v in values]))}


def scoring(pipeline, datapoint):
    result: np.ndarray = pipeline.safe_run(datapoint).predicted_labels_
    reference = datapoint.labels_as_array()

    return {
        "accuracy": accuracy_score(result, reference),
        "per_sample": SingleValueAccuracy((result, reference)),
    }

Now we can run a cross validation. We will only run it on a subset of the data, to keep the runtime manageable.

Note

You might see warnings about retracing of the model. This is because we clone the pipeline before each call to the run method. This is a good idea to ensure that all pipelines are independent of each other, however, might result in some performance overhead.

from tpcp.optimize import Optimize
from tpcp.validate import cross_validate

pipeline = KerasPipeline(n_train_epochs=10)
cv_results = cross_validate(Optimize(pipeline), FashionMNIST()[:100], scoring=scoring, cv=3)

CV Folds:   0%|          | 0/3 [00:00<?, ?it/s](3960, 28, 28)
Epoch 1/10

  1/124 [..............................] - ETA: 1:09 - loss: 2.5074 - accuracy: 0.0938
 22/124 [====>.........................] - ETA: 0s - loss: 1.5346 - accuracy: 0.4389  
 45/124 [=========>....................] - ETA: 0s - loss: 1.2047 - accuracy: 0.5660
 68/124 [===============>..............] - ETA: 0s - loss: 1.0559 - accuracy: 0.6245
 91/124 [=====================>........] - ETA: 0s - loss: 0.9814 - accuracy: 0.6552
114/124 [==========================>...] - ETA: 0s - loss: 0.9195 - accuracy: 0.6806
124/124 [==============================] - 1s 2ms/step - loss: 0.8968 - accuracy: 0.6919
Epoch 2/10

  1/124 [..............................] - ETA: 0s - loss: 1.0481 - accuracy: 0.7500
 24/124 [====>.........................] - ETA: 0s - loss: 0.5817 - accuracy: 0.8112
 46/124 [==========>...................] - ETA: 0s - loss: 0.5914 - accuracy: 0.8023
 68/124 [===============>..............] - ETA: 0s - loss: 0.5709 - accuracy: 0.8065
 90/124 [====================>.........] - ETA: 0s - loss: 0.5670 - accuracy: 0.8087
113/124 [==========================>...] - ETA: 0s - loss: 0.5562 - accuracy: 0.8125
124/124 [==============================] - 0s 2ms/step - loss: 0.5606 - accuracy: 0.8104
Epoch 3/10

  1/124 [..............................] - ETA: 0s - loss: 0.7180 - accuracy: 0.6250
 23/124 [====>.........................] - ETA: 0s - loss: 0.5287 - accuracy: 0.8084
 45/124 [=========>....................] - ETA: 0s - loss: 0.5210 - accuracy: 0.8188
 68/124 [===============>..............] - ETA: 0s - loss: 0.5074 - accuracy: 0.8222
 91/124 [=====================>........] - ETA: 0s - loss: 0.4976 - accuracy: 0.8252
114/124 [==========================>...] - ETA: 0s - loss: 0.4983 - accuracy: 0.8246
124/124 [==============================] - 0s 2ms/step - loss: 0.5009 - accuracy: 0.8232
Epoch 4/10

  1/124 [..............................] - ETA: 0s - loss: 0.4630 - accuracy: 0.8125
 24/124 [====>.........................] - ETA: 0s - loss: 0.4142 - accuracy: 0.8568
 47/124 [==========>...................] - ETA: 0s - loss: 0.4267 - accuracy: 0.8524
 70/124 [===============>..............] - ETA: 0s - loss: 0.4397 - accuracy: 0.8487
 93/124 [=====================>........] - ETA: 0s - loss: 0.4434 - accuracy: 0.8464
116/124 [===========================>..] - ETA: 0s - loss: 0.4432 - accuracy: 0.8443
124/124 [==============================] - 0s 2ms/step - loss: 0.4448 - accuracy: 0.8437
Epoch 5/10

  1/124 [..............................] - ETA: 0s - loss: 0.6924 - accuracy: 0.6250
 24/124 [====>.........................] - ETA: 0s - loss: 0.4531 - accuracy: 0.8216
 47/124 [==========>...................] - ETA: 0s - loss: 0.4399 - accuracy: 0.8384
 70/124 [===============>..............] - ETA: 0s - loss: 0.4194 - accuracy: 0.8473
 93/124 [=====================>........] - ETA: 0s - loss: 0.4076 - accuracy: 0.8548
116/124 [===========================>..] - ETA: 0s - loss: 0.4123 - accuracy: 0.8548
124/124 [==============================] - 0s 2ms/step - loss: 0.4119 - accuracy: 0.8540
Epoch 6/10

  1/124 [..............................] - ETA: 0s - loss: 0.5976 - accuracy: 0.7812
 24/124 [====>.........................] - ETA: 0s - loss: 0.4102 - accuracy: 0.8594
 47/124 [==========>...................] - ETA: 0s - loss: 0.3838 - accuracy: 0.8664
 70/124 [===============>..............] - ETA: 0s - loss: 0.3852 - accuracy: 0.8647
 92/124 [=====================>........] - ETA: 0s - loss: 0.3858 - accuracy: 0.8662
115/124 [==========================>...] - ETA: 0s - loss: 0.3816 - accuracy: 0.8677
124/124 [==============================] - 0s 2ms/step - loss: 0.3792 - accuracy: 0.8687
Epoch 7/10

  1/124 [..............................] - ETA: 0s - loss: 0.4009 - accuracy: 0.8438
 24/124 [====>.........................] - ETA: 0s - loss: 0.3615 - accuracy: 0.8672
 47/124 [==========>...................] - ETA: 0s - loss: 0.3492 - accuracy: 0.8737
 70/124 [===============>..............] - ETA: 0s - loss: 0.3589 - accuracy: 0.8750
 93/124 [=====================>........] - ETA: 0s - loss: 0.3573 - accuracy: 0.8770
116/124 [===========================>..] - ETA: 0s - loss: 0.3569 - accuracy: 0.8772
124/124 [==============================] - 0s 2ms/step - loss: 0.3555 - accuracy: 0.8768
Epoch 8/10

  1/124 [..............................] - ETA: 0s - loss: 0.2810 - accuracy: 0.9062
 24/124 [====>.........................] - ETA: 0s - loss: 0.2964 - accuracy: 0.9128
 47/124 [==========>...................] - ETA: 0s - loss: 0.2937 - accuracy: 0.9049
 70/124 [===============>..............] - ETA: 0s - loss: 0.3136 - accuracy: 0.8955
 93/124 [=====================>........] - ETA: 0s - loss: 0.3299 - accuracy: 0.8901
115/124 [==========================>...] - ETA: 0s - loss: 0.3410 - accuracy: 0.8848
124/124 [==============================] - 0s 2ms/step - loss: 0.3464 - accuracy: 0.8826
Epoch 9/10

  1/124 [..............................] - ETA: 0s - loss: 0.2351 - accuracy: 0.9375
 24/124 [====>.........................] - ETA: 0s - loss: 0.3400 - accuracy: 0.8867
 47/124 [==========>...................] - ETA: 0s - loss: 0.3378 - accuracy: 0.8810
 70/124 [===============>..............] - ETA: 0s - loss: 0.3258 - accuracy: 0.8862
 92/124 [=====================>........] - ETA: 0s - loss: 0.3217 - accuracy: 0.8906
115/124 [==========================>...] - ETA: 0s - loss: 0.3208 - accuracy: 0.8891
124/124 [==============================] - 0s 2ms/step - loss: 0.3196 - accuracy: 0.8886
Epoch 10/10

  1/124 [..............................] - ETA: 0s - loss: 0.2527 - accuracy: 0.9688
 24/124 [====>.........................] - ETA: 0s - loss: 0.2444 - accuracy: 0.9128
 46/124 [==========>...................] - ETA: 0s - loss: 0.2720 - accuracy: 0.9035
 69/124 [===============>..............] - ETA: 0s - loss: 0.2922 - accuracy: 0.8981
 92/124 [=====================>........] - ETA: 0s - loss: 0.2951 - accuracy: 0.8981
115/124 [==========================>...] - ETA: 0s - loss: 0.2948 - accuracy: 0.8992
124/124 [==============================] - 0s 2ms/step - loss: 0.2921 - accuracy: 0.8997


Datapoints:   0%|          | 0/34 [00:00<?, ?it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:   3%|▎         | 1/34 [00:00<00:06,  5.09it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:   6%|▌         | 2/34 [00:00<00:06,  5.05it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:   9%|▉         | 3/34 [00:00<00:05,  5.22it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  12%|█▏        | 4/34 [00:00<00:05,  5.37it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  15%|█▍        | 5/34 [00:00<00:05,  5.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  18%|█▊        | 6/34 [00:01<00:05,  5.36it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  21%|██        | 7/34 [00:01<00:05,  5.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  24%|██▎       | 8/34 [00:01<00:04,  5.30it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  26%|██▋       | 9/34 [00:01<00:04,  5.19it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  29%|██▉       | 10/34 [00:01<00:04,  5.16it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  32%|███▏      | 11/34 [00:02<00:04,  5.27it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  35%|███▌      | 12/34 [00:02<00:04,  5.32it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  38%|███▊      | 13/34 [00:02<00:03,  5.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  41%|████      | 14/34 [00:02<00:03,  5.45it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  44%|████▍     | 15/34 [00:02<00:03,  5.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  47%|████▋     | 16/34 [00:03<00:03,  5.28it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  50%|█████     | 17/34 [00:03<00:03,  5.18it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  53%|█████▎    | 18/34 [00:03<00:03,  5.07it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  56%|█████▌    | 19/34 [00:03<00:02,  5.19it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  59%|█████▉    | 20/34 [00:03<00:02,  5.31it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  62%|██████▏   | 21/34 [00:03<00:02,  5.22it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  65%|██████▍   | 22/34 [00:04<00:02,  5.33it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  68%|██████▊   | 23/34 [00:04<00:02,  5.27it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  71%|███████   | 24/34 [00:04<00:01,  5.15it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  74%|███████▎  | 25/34 [00:04<00:01,  5.13it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  76%|███████▋  | 26/34 [00:04<00:01,  5.24it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  79%|███████▉  | 27/34 [00:05<00:01,  5.13it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  82%|████████▏ | 28/34 [00:05<00:01,  5.22it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  85%|████████▌ | 29/34 [00:05<00:00,  5.31it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  88%|████████▊ | 30/34 [00:05<00:00,  5.16it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  91%|█████████ | 31/34 [00:05<00:00,  5.15it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  94%|█████████▍| 32/34 [00:06<00:00,  5.13it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  97%|█████████▋| 33/34 [00:06<00:00,  5.08it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints: 100%|██████████| 34/34 [00:06<00:00,  5.09it/s]
Datapoints: 100%|██████████| 34/34 [00:06<00:00,  5.23it/s]

CV Folds:  33%|███▎      | 1/3 [00:13<00:26, 13.08s/it](4020, 28, 28)
Epoch 1/10

  1/126 [..............................] - ETA: 1:09 - loss: 2.3656 - accuracy: 0.1250
 23/126 [====>.........................] - ETA: 0s - loss: 1.5016 - accuracy: 0.5014  
 46/126 [=========>....................] - ETA: 0s - loss: 1.2168 - accuracy: 0.5951
 69/126 [===============>..............] - ETA: 0s - loss: 1.0854 - accuracy: 0.6318
 92/126 [====================>.........] - ETA: 0s - loss: 0.9872 - accuracy: 0.6675
115/126 [==========================>...] - ETA: 0s - loss: 0.9161 - accuracy: 0.6902
126/126 [==============================] - 1s 2ms/step - loss: 0.8913 - accuracy: 0.7000
Epoch 2/10

  1/126 [..............................] - ETA: 0s - loss: 0.4984 - accuracy: 0.8125
 23/126 [====>.........................] - ETA: 0s - loss: 0.5577 - accuracy: 0.8043
 46/126 [=========>....................] - ETA: 0s - loss: 0.5677 - accuracy: 0.7989
 69/126 [===============>..............] - ETA: 0s - loss: 0.5895 - accuracy: 0.7912
 92/126 [====================>.........] - ETA: 0s - loss: 0.5754 - accuracy: 0.7959
115/126 [==========================>...] - ETA: 0s - loss: 0.5706 - accuracy: 0.7970
126/126 [==============================] - 0s 2ms/step - loss: 0.5662 - accuracy: 0.8002
Epoch 3/10

  1/126 [..............................] - ETA: 0s - loss: 0.5249 - accuracy: 0.8125
 24/126 [====>.........................] - ETA: 0s - loss: 0.5165 - accuracy: 0.8281
 47/126 [==========>...................] - ETA: 0s - loss: 0.4722 - accuracy: 0.8344
 70/126 [===============>..............] - ETA: 0s - loss: 0.4697 - accuracy: 0.8375
 93/126 [=====================>........] - ETA: 0s - loss: 0.4784 - accuracy: 0.8350
116/126 [==========================>...] - ETA: 0s - loss: 0.4861 - accuracy: 0.8300
126/126 [==============================] - 0s 2ms/step - loss: 0.4893 - accuracy: 0.8279
Epoch 4/10

  1/126 [..............................] - ETA: 0s - loss: 0.2082 - accuracy: 0.9688
 24/126 [====>.........................] - ETA: 0s - loss: 0.4214 - accuracy: 0.8581
 47/126 [==========>...................] - ETA: 0s - loss: 0.4409 - accuracy: 0.8457
 70/126 [===============>..............] - ETA: 0s - loss: 0.4286 - accuracy: 0.8513
 93/126 [=====================>........] - ETA: 0s - loss: 0.4376 - accuracy: 0.8501
116/126 [==========================>...] - ETA: 0s - loss: 0.4469 - accuracy: 0.8443
126/126 [==============================] - 0s 2ms/step - loss: 0.4431 - accuracy: 0.8453
Epoch 5/10

  1/126 [..............................] - ETA: 0s - loss: 0.1979 - accuracy: 0.9375
 24/126 [====>.........................] - ETA: 0s - loss: 0.3937 - accuracy: 0.8620
 47/126 [==========>...................] - ETA: 0s - loss: 0.4028 - accuracy: 0.8531
 70/126 [===============>..............] - ETA: 0s - loss: 0.4030 - accuracy: 0.8567
 93/126 [=====================>........] - ETA: 0s - loss: 0.4077 - accuracy: 0.8558
116/126 [==========================>...] - ETA: 0s - loss: 0.4028 - accuracy: 0.8575
126/126 [==============================] - 0s 2ms/step - loss: 0.4046 - accuracy: 0.8580
Epoch 6/10

  1/126 [..............................] - ETA: 0s - loss: 0.4598 - accuracy: 0.8125
 24/126 [====>.........................] - ETA: 0s - loss: 0.3701 - accuracy: 0.8594
 47/126 [==========>...................] - ETA: 0s - loss: 0.3768 - accuracy: 0.8610
 70/126 [===============>..............] - ETA: 0s - loss: 0.3763 - accuracy: 0.8647
 93/126 [=====================>........] - ETA: 0s - loss: 0.3766 - accuracy: 0.8663
116/126 [==========================>...] - ETA: 0s - loss: 0.3728 - accuracy: 0.8672
126/126 [==============================] - 0s 2ms/step - loss: 0.3772 - accuracy: 0.8667
Epoch 7/10

  1/126 [..............................] - ETA: 0s - loss: 0.2244 - accuracy: 0.9062
 24/126 [====>.........................] - ETA: 0s - loss: 0.3537 - accuracy: 0.8802
 47/126 [==========>...................] - ETA: 0s - loss: 0.3262 - accuracy: 0.8876
 70/126 [===============>..............] - ETA: 0s - loss: 0.3386 - accuracy: 0.8821
 93/126 [=====================>........] - ETA: 0s - loss: 0.3481 - accuracy: 0.8807
116/126 [==========================>...] - ETA: 0s - loss: 0.3472 - accuracy: 0.8790
126/126 [==============================] - 0s 2ms/step - loss: 0.3479 - accuracy: 0.8786
Epoch 8/10

  1/126 [..............................] - ETA: 0s - loss: 0.2092 - accuracy: 0.9375
 24/126 [====>.........................] - ETA: 0s - loss: 0.2914 - accuracy: 0.9049
 47/126 [==========>...................] - ETA: 0s - loss: 0.3042 - accuracy: 0.8956
 70/126 [===============>..............] - ETA: 0s - loss: 0.3183 - accuracy: 0.8879
 93/126 [=====================>........] - ETA: 0s - loss: 0.3303 - accuracy: 0.8864
116/126 [==========================>...] - ETA: 0s - loss: 0.3276 - accuracy: 0.8869
126/126 [==============================] - 0s 2ms/step - loss: 0.3285 - accuracy: 0.8863
Epoch 9/10

  1/126 [..............................] - ETA: 0s - loss: 0.1886 - accuracy: 0.9062
 24/126 [====>.........................] - ETA: 0s - loss: 0.2912 - accuracy: 0.9023
 47/126 [==========>...................] - ETA: 0s - loss: 0.3027 - accuracy: 0.8943
 70/126 [===============>..............] - ETA: 0s - loss: 0.3121 - accuracy: 0.8808
 93/126 [=====================>........] - ETA: 0s - loss: 0.3105 - accuracy: 0.8841
116/126 [==========================>...] - ETA: 0s - loss: 0.3145 - accuracy: 0.8879
126/126 [==============================] - 0s 2ms/step - loss: 0.3120 - accuracy: 0.8900
Epoch 10/10

  1/126 [..............................] - ETA: 0s - loss: 0.2327 - accuracy: 0.9062
 24/126 [====>.........................] - ETA: 0s - loss: 0.2974 - accuracy: 0.8789
 47/126 [==========>...................] - ETA: 0s - loss: 0.3049 - accuracy: 0.8863
 70/126 [===============>..............] - ETA: 0s - loss: 0.3075 - accuracy: 0.8866
 93/126 [=====================>........] - ETA: 0s - loss: 0.3012 - accuracy: 0.8884
116/126 [==========================>...] - ETA: 0s - loss: 0.3039 - accuracy: 0.8887
126/126 [==============================] - 0s 2ms/step - loss: 0.3065 - accuracy: 0.8871


Datapoints:   0%|          | 0/33 [00:00<?, ?it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:   3%|▎         | 1/33 [00:00<00:06,  5.15it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:   6%|▌         | 2/33 [00:00<00:06,  5.05it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:   9%|▉         | 3/33 [00:00<00:05,  5.04it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  12%|█▏        | 4/33 [00:00<00:05,  5.21it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  15%|█▌        | 5/33 [00:00<00:05,  5.33it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  18%|█▊        | 6/33 [00:01<00:05,  5.27it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  21%|██        | 7/33 [00:01<00:05,  5.17it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  24%|██▍       | 8/33 [00:01<00:04,  5.12it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  27%|██▋       | 9/33 [00:01<00:04,  5.24it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  30%|███       | 10/33 [00:01<00:04,  5.12it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  33%|███▎      | 11/33 [00:02<00:04,  5.09it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  36%|███▋      | 12/33 [00:02<00:04,  5.06it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  39%|███▉      | 13/33 [00:02<00:03,  5.06it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  42%|████▏     | 14/33 [00:02<00:03,  5.05it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  45%|████▌     | 15/33 [00:02<00:03,  5.04it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  48%|████▊     | 16/33 [00:03<00:03,  5.00it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  52%|█████▏    | 17/33 [00:03<00:03,  5.13it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  55%|█████▍    | 18/33 [00:03<00:02,  5.24it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  58%|█████▊    | 19/33 [00:03<00:02,  5.27it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  61%|██████    | 20/33 [00:03<00:02,  5.21it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  64%|██████▎   | 21/33 [00:04<00:02,  5.17it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  67%|██████▋   | 22/33 [00:04<00:02,  5.06it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  70%|██████▉   | 23/33 [00:04<00:01,  5.15it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  73%|███████▎  | 24/33 [00:04<00:01,  5.23it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  76%|███████▌  | 25/33 [00:04<00:01,  5.28it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  79%|███████▉  | 26/33 [00:05<00:01,  5.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  82%|████████▏ | 27/33 [00:05<00:01,  5.29it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  85%|████████▍ | 28/33 [00:05<00:00,  5.19it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  88%|████████▊ | 29/33 [00:05<00:00,  5.16it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  91%|█████████ | 30/33 [00:05<00:00,  5.16it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  94%|█████████▍| 31/33 [00:06<00:00,  5.11it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  97%|█████████▋| 32/33 [00:06<00:00,  5.23it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints: 100%|██████████| 33/33 [00:06<00:00,  5.32it/s]
Datapoints: 100%|██████████| 33/33 [00:06<00:00,  5.18it/s]

CV Folds:  67%|██████▋   | 2/3 [00:23<00:11, 11.65s/it](4020, 28, 28)
Epoch 1/10

  1/126 [..............................] - ETA: 1:08 - loss: 2.3129 - accuracy: 0.0938
 24/126 [====>.........................] - ETA: 0s - loss: 1.4718 - accuracy: 0.5026  
 47/126 [==========>...................] - ETA: 0s - loss: 1.2165 - accuracy: 0.5904
 69/126 [===============>..............] - ETA: 0s - loss: 1.0876 - accuracy: 0.6322
 92/126 [====================>.........] - ETA: 0s - loss: 0.9881 - accuracy: 0.6678
115/126 [==========================>...] - ETA: 0s - loss: 0.9221 - accuracy: 0.6894
126/126 [==============================] - 1s 2ms/step - loss: 0.8965 - accuracy: 0.6983
Epoch 2/10

  1/126 [..............................] - ETA: 0s - loss: 0.4769 - accuracy: 0.8125
 24/126 [====>.........................] - ETA: 0s - loss: 0.5971 - accuracy: 0.7930
 47/126 [==========>...................] - ETA: 0s - loss: 0.5900 - accuracy: 0.7992
 70/126 [===============>..............] - ETA: 0s - loss: 0.5994 - accuracy: 0.7929
 93/126 [=====================>........] - ETA: 0s - loss: 0.5830 - accuracy: 0.8017
116/126 [==========================>...] - ETA: 0s - loss: 0.5656 - accuracy: 0.8060
126/126 [==============================] - 0s 2ms/step - loss: 0.5750 - accuracy: 0.8040
Epoch 3/10

  1/126 [..............................] - ETA: 0s - loss: 0.6893 - accuracy: 0.7812
 24/126 [====>.........................] - ETA: 0s - loss: 0.5626 - accuracy: 0.8060
 47/126 [==========>...................] - ETA: 0s - loss: 0.5343 - accuracy: 0.8145
 69/126 [===============>..............] - ETA: 0s - loss: 0.5346 - accuracy: 0.8134
 91/126 [====================>.........] - ETA: 0s - loss: 0.5219 - accuracy: 0.8218
113/126 [=========================>....] - ETA: 0s - loss: 0.5104 - accuracy: 0.8252
126/126 [==============================] - 0s 2ms/step - loss: 0.5140 - accuracy: 0.8224
Epoch 4/10

  1/126 [..............................] - ETA: 0s - loss: 0.4519 - accuracy: 0.9062
 24/126 [====>.........................] - ETA: 0s - loss: 0.4256 - accuracy: 0.8581
 47/126 [==========>...................] - ETA: 0s - loss: 0.4503 - accuracy: 0.8424
 70/126 [===============>..............] - ETA: 0s - loss: 0.4453 - accuracy: 0.8415
 93/126 [=====================>........] - ETA: 0s - loss: 0.4420 - accuracy: 0.8444
116/126 [==========================>...] - ETA: 0s - loss: 0.4631 - accuracy: 0.8384
126/126 [==============================] - 0s 2ms/step - loss: 0.4667 - accuracy: 0.8358
Epoch 5/10

  1/126 [..............................] - ETA: 0s - loss: 0.4192 - accuracy: 0.8125
 24/126 [====>.........................] - ETA: 0s - loss: 0.4185 - accuracy: 0.8633
 47/126 [==========>...................] - ETA: 0s - loss: 0.4139 - accuracy: 0.8630
 70/126 [===============>..............] - ETA: 0s - loss: 0.4153 - accuracy: 0.8643
 93/126 [=====================>........] - ETA: 0s - loss: 0.4228 - accuracy: 0.8599
116/126 [==========================>...] - ETA: 0s - loss: 0.4171 - accuracy: 0.8610
126/126 [==============================] - 0s 2ms/step - loss: 0.4173 - accuracy: 0.8600
Epoch 6/10

  1/126 [..............................] - ETA: 0s - loss: 0.8799 - accuracy: 0.7500
 24/126 [====>.........................] - ETA: 0s - loss: 0.3761 - accuracy: 0.8672
 47/126 [==========>...................] - ETA: 0s - loss: 0.3851 - accuracy: 0.8677
 70/126 [===============>..............] - ETA: 0s - loss: 0.3909 - accuracy: 0.8661
 93/126 [=====================>........] - ETA: 0s - loss: 0.3953 - accuracy: 0.8616
116/126 [==========================>...] - ETA: 0s - loss: 0.3941 - accuracy: 0.8631
126/126 [==============================] - 0s 2ms/step - loss: 0.3949 - accuracy: 0.8624
Epoch 7/10

  1/126 [..............................] - ETA: 0s - loss: 0.2022 - accuracy: 0.9375
 24/126 [====>.........................] - ETA: 0s - loss: 0.3764 - accuracy: 0.8776
 47/126 [==========>...................] - ETA: 0s - loss: 0.3475 - accuracy: 0.8816
 70/126 [===============>..............] - ETA: 0s - loss: 0.3481 - accuracy: 0.8781
 93/126 [=====================>........] - ETA: 0s - loss: 0.3624 - accuracy: 0.8774
115/126 [==========================>...] - ETA: 0s - loss: 0.3652 - accuracy: 0.8764
126/126 [==============================] - 0s 2ms/step - loss: 0.3635 - accuracy: 0.8774
Epoch 8/10

  1/126 [..............................] - ETA: 0s - loss: 0.3186 - accuracy: 0.8750
 24/126 [====>.........................] - ETA: 0s - loss: 0.3329 - accuracy: 0.8841
 47/126 [==========>...................] - ETA: 0s - loss: 0.3320 - accuracy: 0.8856
 70/126 [===============>..............] - ETA: 0s - loss: 0.3330 - accuracy: 0.8862
 93/126 [=====================>........] - ETA: 0s - loss: 0.3369 - accuracy: 0.8858
116/126 [==========================>...] - ETA: 0s - loss: 0.3402 - accuracy: 0.8825
126/126 [==============================] - 0s 2ms/step - loss: 0.3457 - accuracy: 0.8826
Epoch 9/10

  1/126 [..............................] - ETA: 0s - loss: 0.2176 - accuracy: 0.9375
 24/126 [====>.........................] - ETA: 0s - loss: 0.3504 - accuracy: 0.8828
 47/126 [==========>...................] - ETA: 0s - loss: 0.3412 - accuracy: 0.8763
 70/126 [===============>..............] - ETA: 0s - loss: 0.3244 - accuracy: 0.8844
 93/126 [=====================>........] - ETA: 0s - loss: 0.3284 - accuracy: 0.8837
116/126 [==========================>...] - ETA: 0s - loss: 0.3302 - accuracy: 0.8885
126/126 [==============================] - 0s 2ms/step - loss: 0.3285 - accuracy: 0.8878
Epoch 10/10

  1/126 [..............................] - ETA: 0s - loss: 0.3079 - accuracy: 0.8125
 23/126 [====>.........................] - ETA: 0s - loss: 0.2855 - accuracy: 0.9049
 46/126 [=========>....................] - ETA: 0s - loss: 0.2977 - accuracy: 0.9029
 68/126 [===============>..............] - ETA: 0s - loss: 0.2887 - accuracy: 0.9053
 90/126 [====================>.........] - ETA: 0s - loss: 0.2920 - accuracy: 0.9049
112/126 [=========================>....] - ETA: 0s - loss: 0.2976 - accuracy: 0.9004
126/126 [==============================] - 0s 2ms/step - loss: 0.3008 - accuracy: 0.8993


Datapoints:   0%|          | 0/33 [00:00<?, ?it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:   3%|▎         | 1/33 [00:00<00:05,  5.61it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:   6%|▌         | 2/33 [00:00<00:05,  5.60it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:   9%|▉         | 3/33 [00:00<00:05,  5.55it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  12%|█▏        | 4/33 [00:00<00:05,  5.48it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  15%|█▌        | 5/33 [00:00<00:05,  5.52it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  18%|█▊        | 6/33 [00:01<00:04,  5.55it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  21%|██        | 7/33 [00:01<00:04,  5.35it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  24%|██▍       | 8/33 [00:01<00:04,  5.26it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  27%|██▋       | 9/33 [00:01<00:04,  5.31it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  30%|███       | 10/33 [00:01<00:04,  5.37it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  33%|███▎      | 11/33 [00:02<00:04,  5.45it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  36%|███▋      | 12/33 [00:02<00:03,  5.34it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  39%|███▉      | 13/33 [00:02<00:03,  5.24it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  42%|████▏     | 14/33 [00:02<00:03,  5.20it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  45%|████▌     | 15/33 [00:02<00:03,  5.15it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  48%|████▊     | 16/33 [00:03<00:03,  5.11it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  52%|█████▏    | 17/33 [00:03<00:03,  5.12it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  55%|█████▍    | 18/33 [00:03<00:02,  5.11it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  58%|█████▊    | 19/33 [00:03<00:02,  5.18it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  61%|██████    | 20/33 [00:03<00:02,  5.30it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  64%|██████▎   | 21/33 [00:03<00:02,  5.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  67%|██████▋   | 22/33 [00:04<00:02,  5.26it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  70%|██████▉   | 23/33 [00:04<00:01,  5.37it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  73%|███████▎  | 24/33 [00:04<00:01,  5.30it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  76%|███████▌  | 25/33 [00:04<00:01,  5.34it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  79%|███████▉  | 26/33 [00:04<00:01,  5.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  82%|████████▏ | 27/33 [00:05<00:01,  5.24it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  85%|████████▍ | 28/33 [00:05<00:00,  5.11it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  88%|████████▊ | 29/33 [00:05<00:00,  5.21it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  91%|█████████ | 30/33 [00:05<00:00,  5.30it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  94%|█████████▍| 31/33 [00:06<00:00,  3.23it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  97%|█████████▋| 32/33 [00:06<00:00,  3.62it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints: 100%|██████████| 33/33 [00:06<00:00,  4.00it/s]
Datapoints: 100%|██████████| 33/33 [00:06<00:00,  4.97it/s]

CV Folds: 100%|██████████| 3/3 [00:36<00:00, 12.34s/it]
CV Folds: 100%|██████████| 3/3 [00:36<00:00, 12.30s/it]

We can now look at the results per group:

cv_results["test_single_accuracy"]

[[0.85, 0.8666666666666667, 0.9333333333333333, 0.8, 0.85, 0.85, 0.85, 0.8833333333333333, 0.85, 0.85, 0.8166666666666667, 0.8666666666666667, 0.9333333333333333, 0.8, 0.8833333333333333, 0.8, 0.8, 0.8833333333333333, 0.7833333333333333, 0.8166666666666667, 0.8, 0.85, 0.8, 0.95, 0.8, 0.8833333333333333, 0.8833333333333333, 0.85, 0.85, 0.7833333333333333, 0.75, 0.8666666666666667, 0.8333333333333334, 0.9333333333333333], [0.7833333333333333, 0.8166666666666667, 0.7666666666666667, 0.7833333333333333, 0.85, 0.8, 0.8, 0.8166666666666667, 0.8, 0.8333333333333334, 0.8333333333333334, 0.8, 0.8166666666666667, 0.85, 0.75, 0.7, 0.8333333333333334, 0.8166666666666667, 0.8666666666666667, 0.7666666666666667, 0.7833333333333333, 0.7833333333333333, 0.7666666666666667, 0.8833333333333333, 0.8333333333333334, 0.7833333333333333, 0.8, 0.85, 0.85, 0.8166666666666667, 0.7666666666666667, 0.8, 0.8833333333333333], [0.7666666666666667, 0.9166666666666666, 0.8, 0.8, 0.7833333333333333, 0.85, 0.8833333333333333, 0.85, 0.9, 0.8833333333333333, 0.9, 0.85, 0.8833333333333333, 0.8166666666666667, 0.8333333333333334, 0.8333333333333334, 0.85, 0.7833333333333333, 0.7166666666666667, 0.7333333333333333, 0.75, 0.7833333333333333, 0.85, 0.7666666666666667, 0.7833333333333333, 0.9, 0.8666666666666667, 0.8333333333333334, 0.8333333333333334, 0.8166666666666667, 0.85, 0.85, 0.9]]

And the overall accuracy as the average over all samples of all groups within a fold:

cv_results["test_per_sample__accuracy"]

array([0.84705882, 0.80858586, 0.83080808])

Total running time of the script: (0 minutes 41.496 seconds)

Estimated memory usage: 284 MB

Gallery generated by Sphinx-Gallery