Note

Go to the end to download the full example code

Tensorflow/Keras#

Note

This example requires the tensorflow package to be installed.

Theoretically, tpcp is framework agnostic and can be used with any framework. However, due to the way some frameworks handle their objects, some special handling internally is required. Hence, this example does not only serve as example on how to use tensorflow with tpcp, but also as a test case for these special cases.

When using tpcp with any machine learning framework, you either want to use a pretrained model with a normal pipeline or a train your own model as part of an Optimizable Pipeline. Here we show the second case, as it is more complex, and you are likely able to figure out the first case yourself.

This means, we are planning to perform the following steps:

  1. Create a pipeline that creates and trains a model.

  2. Allow the modification of model hyperparameters.

  3. Run a simple cross-validation to demonstrate the functionality.

This example reimplements the basic MNIST example from the [tensorflow documentation](https://www.tensorflow.org/tutorials/keras/classification).

Some Notes#

In this example we show how to implement a Pipeline that uses tensorflow. You could implement an Algorithm in a similar way. This would actually be easier, as no specific handling of the input data would be required. For a pipeline, we need to create a custom Dataset class, as this is the expected input for a pipeline.

The Dataset#

We are using the normal fashion MNIST dataset for this example It consists of 60.000 images of 28x28 pixels, each with a label. We will ignore the typical train-test split, as we want to do our own cross-validation.

In addition, we will simulate an additional “index level”. In this (and most typical deep learning datasets), each datapoint is one vector for which we can make one prediction. In tpcp, we usually deal with datasets, where you might have multiple pieces of information for each datapoint. For example, one datapoint could be a patient, for which we have an entire time series of measurements. We will simulate this here, by creating the index of our dataset as 1000 groups each containing 60 images.

Other than that, the dataset is pretty standard. Besides the create_index method, we only need to implement the input_as_array and labels_as_array methods that allow us to easily access the data once we selected a single group.

from functools import lru_cache

import numpy as np
import pandas as pd
import tensorflow as tf

from tpcp import Dataset

tf.keras.utils.set_random_seed(812)
tf.config.experimental.enable_op_determinism()


@lru_cache(maxsize=1)
def get_fashion_mnist_data():
    # Note: We throw train and test sets together, as we don't care about the official split here.
    #       We will create our own split later.
    (train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.fashion_mnist.load_data()
    return np.array(list(train_images) + list(test_images)), list(train_labels) + list(test_labels)


class FashionMNIST(Dataset):
    def input_as_array(self) -> np.ndarray:
        self.assert_is_single(None, "input_as_array")
        group_id = int(self.group_label.group_id)
        images, _ = get_fashion_mnist_data()
        return images[group_id * 60 : (group_id + 1) * 60].reshape((60, 28, 28)) / 255

    def labels_as_array(self) -> np.ndarray:
        self.assert_is_single(None, "labels_as_array")
        group_id = int(self.group_label.group_id)
        _, labels = get_fashion_mnist_data()
        return np.array(labels[group_id * 60 : (group_id + 1) * 60])

    def create_index(self) -> pd.DataFrame:
        # There are 60.000 images in total.
        # We simulate 1000 groups of 60 images each.
        return pd.DataFrame({"group_id": list(range(1000))})

We can see our Dataset works as expected:

dataset = FashionMNIST()
dataset[0].input_as_array().shape

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz

    0/29515 ━━━━━━━━━━━━━━━━━━━━ 0s 0s/step
29515/29515 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz

       0/26421880 ━━━━━━━━━━━━━━━━━━━━ 0s 0s/step
  327680/26421880 ━━━━━━━━━━━━━━━━━━━━ 4s 0us/step
 6299648/26421880 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
15892480/26421880 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
25452544/26421880 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
26421880/26421880 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz

   0/5148 ━━━━━━━━━━━━━━━━━━━━ 0s 0s/step
5148/5148 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz

      0/4422102 ━━━━━━━━━━━━━━━━━━━━ 0s 0s/step
 868352/4422102 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
4422102/4422102 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step

(60, 28, 28)

dataset[0].labels_as_array().shape

(60,)

The Pipeline#

We will create a pipeline that uses a simple neural network to classify the images. In tpcp, all “things” that should be optimized need to be parameters. This means our model itself needs to be a parameter of the pipeline. However, as we don’t have the model yet, as its creation depends on other hyperparameters, we add it as an optional parameter initialized with None. Further, we prefix the parameter name with an underscore, to signify, that this is not a parameter that should be modified manually by the user. This is just convention, and it is up to you to decide how you want to name your parameters.

We further introduce a hyperparameter n_dense_layer_nodes to show how we can influence the model creation.

The optimize method#

To make our pipeline optimizable, it needs to inherit from OptimizablePipeline. Further we need to mark at least one of the parameters as OptiPara using the type annotation. We do this for our _model parameter.

Finally, we need to implement the self_optimize method. This method will get the entire training dataset as input and should update the _model parameter with the trained model. Hence, we first extract the relevant data (remember, each datapoint is 60 images), by concatinating all images over all groups in the dataset. Then we create the Keras model based on the hyperparameters. Finally, we train the model and update the _model parameter.

Here we chose to wrap the method with make_optimize_safe. This decorator will perform some runtime checks to ensure that the method is implemented correctly.

The run method#

The run method expects that the _model parameter is already set (i.e. the pipeline was already optimized). It gets a single datapoint as input (remember, a datapoint is a single group of 60 images). We then extract the data from the datapoint and let the model make a prediction. We store the prediction on our output attribute predictions_. The trailing underscore is a convention to signify, that this is an “result” attribute.

import warnings
from typing import Optional

from typing_extensions import Self

from tpcp import OptimizablePipeline, OptiPara, make_action_safe, make_optimize_safe


class KerasPipeline(OptimizablePipeline):
    n_dense_layer_nodes: int
    n_train_epochs: int
    _model: OptiPara[Optional[tf.keras.Sequential]]

    predictions_: np.ndarray

    def __init__(self, n_dense_layer_nodes=128, n_train_epochs=5, _model: Optional[tf.keras.Sequential] = None):
        self.n_dense_layer_nodes = n_dense_layer_nodes
        self.n_train_epochs = n_train_epochs
        self._model = _model

    @property
    def predicted_labels_(self):
        return np.argmax(self.predictions_, axis=1)

    @make_optimize_safe
    def self_optimize(self, dataset, **_) -> Self:
        data = tf.convert_to_tensor(np.vstack([d.input_as_array() for d in dataset]))
        labels = tf.convert_to_tensor(np.hstack([d.labels_as_array() for d in dataset]))

        print(data.shape)
        if self._model is not None:
            warnings.warn("Overwriting existing model!")

        self._model = tf.keras.Sequential(
            [
                tf.keras.layers.Input((28, 28)),
                tf.keras.layers.Flatten(),
                tf.keras.layers.Dense(self.n_dense_layer_nodes, activation="relu"),
                tf.keras.layers.Dense(10),
            ]
        )

        self._model.compile(
            optimizer="adam",
            loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
            metrics=["accuracy"],
        )

        self._model.fit(data, labels, epochs=self.n_train_epochs)

        return self

    @make_action_safe
    def run(self, datapoint) -> Self:
        if self._model is None:
            raise RuntimeError("Model not trained yet!")
        data = tf.convert_to_tensor(datapoint.input_as_array())

        self.predictions_ = self._model.predict(data)
        return self

Testing the pipeline#

We can now test our pipeline. We will run the optimization using a couple of datapoints (to keep everything fast) and then use run to get the predictions for a single unseen datapoint.

pipeline = KerasPipeline().self_optimize(FashionMNIST()[:10])
p1 = pipeline.run(FashionMNIST()[11])
print(p1.predicted_labels_)
print(FashionMNIST()[11].labels_as_array())

(600, 28, 28)
Epoch 1/5

 1/19 ━━━━━━━━━━━━━━━━━━━━ 15s 840ms/step - accuracy: 0.1250 - loss: 2.3593
19/19 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.3623 - loss: 1.8187
Epoch 2/5

 1/19 ━━━━━━━━━━━━━━━━━━━━ 0s 55ms/step - accuracy: 0.7188 - loss: 1.0003
19/19 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.6958 - loss: 0.9131
Epoch 3/5

 1/19 ━━━━━━━━━━━━━━━━━━━━ 0s 19ms/step - accuracy: 0.7812 - loss: 0.7427
19/19 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7586 - loss: 0.7140
Epoch 4/5

 1/19 ━━━━━━━━━━━━━━━━━━━━ 0s 18ms/step - accuracy: 0.7812 - loss: 0.6089
19/19 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7986 - loss: 0.5833
Epoch 5/5

 1/19 ━━━━━━━━━━━━━━━━━━━━ 0s 18ms/step - accuracy: 0.7812 - loss: 0.5329
19/19 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8290 - loss: 0.5055

1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 46ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
[8 8 6 9 4 0 7 3 7 9 4 8 4 3 7 8 1 4 0 7 9 8 5 5 2 1 3 4 1 9 7 5 9 9 7 8 2
 7 4 7 2 4 7 1 1 7 5 4 8 3 5 9 0 7 4 0 0 9 1 9]
[8 8 0 9 2 0 7 3 7 9 3 8 4 3 7 8 1 4 0 7 9 8 5 5 2 1 3 4 6 7 7 5 9 9 7 8 2
 7 4 7 0 3 5 1 1 5 5 2 8 3 5 9 0 7 3 0 0 7 1 9]

We can see that even with just 5 epochs, the model already performs quite well. To quantify we can calculate the accuracy for this datapoint:

from sklearn.metrics import accuracy_score

accuracy_score(p1.predicted_labels_, FashionMNIST()[11].labels_as_array())

0.8

Cross Validation#

If we want to run a cross validation, we need to formalize the scoring into a function. We will calculate two types of accuracy: First, the accuracy per group and second, the accuracy over all images across all groups. For more information about how this works, check the Custom Scorer example.

from collections.abc import Sequence

from tpcp.validate import Aggregator


class SingleValueAccuracy(Aggregator[np.ndarray]):
    RETURN_RAW_SCORES = False

    @classmethod
    def aggregate(cls, /, values: Sequence[tuple[np.ndarray, np.ndarray]], **_) -> dict[str, float]:
        return {"accuracy": accuracy_score(np.hstack([v[0] for v in values]), np.hstack([v[1] for v in values]))}


def scoring(pipeline, datapoint):
    result: np.ndarray = pipeline.safe_run(datapoint).predicted_labels_
    reference = datapoint.labels_as_array()

    return {
        "accuracy": accuracy_score(result, reference),
        "per_sample": SingleValueAccuracy((result, reference)),
    }

Now we can run a cross validation. We will only run it on a subset of the data, to keep the runtime manageable.

Note

You might see warnings about retracing of the model. This is because we clone the pipeline before each call to the run method. This is a good idea to ensure that all pipelines are independent of each other, however, might result in some performance overhead.

from tpcp.optimize import Optimize
from tpcp.validate import cross_validate

pipeline = KerasPipeline(n_train_epochs=10)
cv_results = cross_validate(Optimize(pipeline), FashionMNIST()[:100], scoring=scoring, cv=3)

CV Folds:   0%|          | 0/3 [00:00<?, ?it/s](3960, 28, 28)
Epoch 1/10

  1/124 ━━━━━━━━━━━━━━━━━━━━ 1:37 790ms/step - accuracy: 0.0938 - loss: 2.5074
 23/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.3104 - loss: 1.8620    
 47/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.4182 - loss: 1.5844
 71/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.4827 - loss: 1.4197
 94/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.5227 - loss: 1.3170
118/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.5534 - loss: 1.2396
124/124 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.5611 - loss: 1.2206
Epoch 2/10

  1/124 ━━━━━━━━━━━━━━━━━━━━ 3s 27ms/step - accuracy: 0.8750 - loss: 0.4848
 24/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7849 - loss: 0.6267 
 48/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7893 - loss: 0.6098
 72/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7937 - loss: 0.5973
 96/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7961 - loss: 0.5909
120/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7985 - loss: 0.5869
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7990 - loss: 0.5862
Epoch 3/10

  1/124 ━━━━━━━━━━━━━━━━━━━━ 4s 38ms/step - accuracy: 0.8438 - loss: 0.4153
 25/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8117 - loss: 0.5263 
 49/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8215 - loss: 0.5120
 72/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8260 - loss: 0.5045
 96/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8278 - loss: 0.5015
120/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8291 - loss: 0.5003
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8293 - loss: 0.5000
Epoch 4/10

  1/124 ━━━━━━━━━━━━━━━━━━━━ 3s 32ms/step - accuracy: 0.8750 - loss: 0.3588
 24/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8290 - loss: 0.4597 
 49/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8370 - loss: 0.4517
 73/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8397 - loss: 0.4485
 98/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8407 - loss: 0.4486
123/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8420 - loss: 0.4488
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8421 - loss: 0.4488
Epoch 5/10

  1/124 ━━━━━━━━━━━━━━━━━━━━ 2s 17ms/step - accuracy: 0.8750 - loss: 0.3387
 25/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8449 - loss: 0.4166 
 48/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8510 - loss: 0.4119
 71/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8529 - loss: 0.4107
 95/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8541 - loss: 0.4112
119/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8554 - loss: 0.4118
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8557 - loss: 0.4119
Epoch 6/10

  1/124 ━━━━━━━━━━━━━━━━━━━━ 2s 19ms/step - accuracy: 0.8750 - loss: 0.3044
 25/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8584 - loss: 0.3748 
 49/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8627 - loss: 0.3748
 72/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8629 - loss: 0.3769
 95/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8630 - loss: 0.3790
119/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8636 - loss: 0.3806
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8638 - loss: 0.3807
Epoch 7/10

  1/124 ━━━━━━━━━━━━━━━━━━━━ 3s 32ms/step - accuracy: 0.9062 - loss: 0.2879
 25/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8604 - loss: 0.3449 
 49/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8654 - loss: 0.3469
 73/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8669 - loss: 0.3503
 97/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8679 - loss: 0.3527
121/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8690 - loss: 0.3541
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8692 - loss: 0.3543
Epoch 8/10

  1/124 ━━━━━━━━━━━━━━━━━━━━ 4s 35ms/step - accuracy: 0.8750 - loss: 0.2663
 25/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8672 - loss: 0.3189 
 50/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8731 - loss: 0.3232
 74/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8754 - loss: 0.3274
 99/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8769 - loss: 0.3301
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8780 - loss: 0.3317
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8780 - loss: 0.3317
Epoch 9/10

  1/124 ━━━━━━━━━━━━━━━━━━━━ 2s 17ms/step - accuracy: 0.9375 - loss: 0.2468
 26/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8889 - loss: 0.2957 
 50/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8886 - loss: 0.3004
 75/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8880 - loss: 0.3048
100/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8879 - loss: 0.3077
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8881 - loss: 0.3095
Epoch 10/10

  1/124 ━━━━━━━━━━━━━━━━━━━━ 2s 18ms/step - accuracy: 0.9375 - loss: 0.2620
 25/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8991 - loss: 0.2781 
 49/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8973 - loss: 0.2828
 73/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8961 - loss: 0.2871
 97/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8953 - loss: 0.2898
121/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8950 - loss: 0.2915
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8950 - loss: 0.2917


Datapoints:   0%|          | 0/34 [00:00<?, ?it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:   3%|▎         | 1/34 [00:00<00:04,  7.10it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:   6%|▌         | 2/34 [00:00<00:04,  7.05it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step


Datapoints:   9%|▉         | 3/34 [00:00<00:04,  6.98it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  12%|█▏        | 4/34 [00:00<00:04,  6.96it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 41ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 20ms/step


Datapoints:  15%|█▍        | 5/34 [00:00<00:04,  7.18it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 42ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 20ms/step


Datapoints:  18%|█▊        | 6/34 [00:00<00:03,  7.33it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 41ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 20ms/step


Datapoints:  21%|██        | 7/34 [00:00<00:03,  7.41it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 40ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 20ms/step


Datapoints:  24%|██▎       | 8/34 [00:01<00:03,  7.45it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 41ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  26%|██▋       | 9/34 [00:01<00:03,  7.47it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 41ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step


Datapoints:  29%|██▉       | 10/34 [00:01<00:03,  7.47it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step


Datapoints:  32%|███▏      | 11/34 [00:01<00:03,  7.45it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 42ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step


Datapoints:  35%|███▌      | 12/34 [00:01<00:02,  7.43it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 41ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step


Datapoints:  38%|███▊      | 13/34 [00:01<00:02,  7.42it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 42ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  41%|████      | 14/34 [00:01<00:02,  7.38it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 42ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step


Datapoints:  44%|████▍     | 15/34 [00:02<00:02,  7.30it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  47%|████▋     | 16/34 [00:02<00:02,  7.35it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step


Datapoints:  50%|█████     | 17/34 [00:02<00:02,  7.33it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step


Datapoints:  53%|█████▎    | 18/34 [00:02<00:02,  7.26it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  56%|█████▌    | 19/34 [00:02<00:02,  7.24it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  59%|█████▉    | 20/34 [00:02<00:01,  7.23it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  62%|██████▏   | 21/34 [00:02<00:01,  7.19it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 48ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  65%|██████▍   | 22/34 [00:03<00:01,  7.10it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  68%|██████▊   | 23/34 [00:03<00:01,  7.09it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  71%|███████   | 24/34 [00:03<00:01,  7.11it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step


Datapoints:  74%|███████▎  | 25/34 [00:03<00:01,  7.18it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  76%|███████▋  | 26/34 [00:03<00:01,  7.03it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  79%|███████▉  | 27/34 [00:03<00:00,  7.10it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  82%|████████▏ | 28/34 [00:03<00:00,  7.11it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  85%|████████▌ | 29/34 [00:04<00:00,  7.12it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  88%|████████▊ | 30/34 [00:04<00:00,  7.07it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  91%|█████████ | 31/34 [00:04<00:00,  7.13it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  94%|█████████▍| 32/34 [00:04<00:00,  7.04it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 47ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step


Datapoints:  97%|█████████▋| 33/34 [00:04<00:00,  7.01it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step


Datapoints: 100%|██████████| 34/34 [00:04<00:00,  7.11it/s]
Datapoints: 100%|██████████| 34/34 [00:04<00:00,  7.21it/s]

CV Folds:  33%|███▎      | 1/3 [00:09<00:18,  9.09s/it](4020, 28, 28)
Epoch 1/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 1:36 775ms/step - accuracy: 0.1250 - loss: 2.3656
 25/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.3601 - loss: 1.8376    
 49/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.4608 - loss: 1.5763
 73/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.5132 - loss: 1.4284
 96/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.5481 - loss: 1.3290
120/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.5752 - loss: 1.2504
126/126 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.5820 - loss: 1.2309
Epoch 2/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 3s 30ms/step - accuracy: 0.8438 - loss: 0.4830
 23/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7652 - loss: 0.6225 
 46/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7661 - loss: 0.6237
 70/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7698 - loss: 0.6191
 93/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7739 - loss: 0.6137
116/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7773 - loss: 0.6078
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7789 - loss: 0.6047
Epoch 3/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 2s 19ms/step - accuracy: 0.8750 - loss: 0.3561
 24/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8127 - loss: 0.5169 
 48/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8123 - loss: 0.5232
 73/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8137 - loss: 0.5215
 97/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8155 - loss: 0.5190
121/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8172 - loss: 0.5157
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8177 - loss: 0.5147
Epoch 4/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 4s 33ms/step - accuracy: 0.9375 - loss: 0.3082
 25/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8386 - loss: 0.4719 
 49/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8341 - loss: 0.4832
 73/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8340 - loss: 0.4819
 97/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8349 - loss: 0.4797
121/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8362 - loss: 0.4763
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8366 - loss: 0.4753
Epoch 5/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 4s 32ms/step - accuracy: 0.8750 - loss: 0.3008
 25/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8340 - loss: 0.4339 
 49/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8364 - loss: 0.4400
 73/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8395 - loss: 0.4370
 97/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8418 - loss: 0.4350
121/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8440 - loss: 0.4323
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8446 - loss: 0.4315
Epoch 6/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 2s 18ms/step - accuracy: 0.8750 - loss: 0.2896
 25/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8394 - loss: 0.4028 
 49/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8437 - loss: 0.4081
 74/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8488 - loss: 0.4049
 97/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8517 - loss: 0.4033
120/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8540 - loss: 0.4010
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8547 - loss: 0.4001
Epoch 7/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 2s 19ms/step - accuracy: 0.8750 - loss: 0.2921
 24/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8348 - loss: 0.3770 
 48/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8409 - loss: 0.3816
 72/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8477 - loss: 0.3784
 96/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8519 - loss: 0.3765
120/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8552 - loss: 0.3743
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8561 - loss: 0.3735
Epoch 8/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 2s 18ms/step - accuracy: 0.8750 - loss: 0.2784
 25/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8476 - loss: 0.3531 
 49/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8527 - loss: 0.3575
 72/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8580 - loss: 0.3550
 94/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8615 - loss: 0.3536
117/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8643 - loss: 0.3519
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8655 - loss: 0.3509
Epoch 9/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 2s 23ms/step - accuracy: 0.8750 - loss: 0.2839
 26/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8575 - loss: 0.3364 
 51/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8617 - loss: 0.3381
 75/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8667 - loss: 0.3344
 99/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8702 - loss: 0.3327
123/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8729 - loss: 0.3308
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8734 - loss: 0.3304
Epoch 10/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 2s 18ms/step - accuracy: 0.8750 - loss: 0.2753
 24/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8615 - loss: 0.3161 
 48/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8675 - loss: 0.3182
 72/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8733 - loss: 0.3150
 96/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8773 - loss: 0.3134
120/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8800 - loss: 0.3118
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8808 - loss: 0.3112


Datapoints:   0%|          | 0/33 [00:00<?, ?it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:   3%|▎         | 1/33 [00:00<00:04,  7.07it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:   6%|▌         | 2/33 [00:00<00:04,  7.03it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 47ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:   9%|▉         | 3/33 [00:00<00:04,  6.99it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step


Datapoints:  12%|█▏        | 4/33 [00:00<00:04,  7.09it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 42ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  15%|█▌        | 5/33 [00:00<00:03,  7.16it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  18%|█▊        | 6/33 [00:00<00:03,  7.15it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step


Datapoints:  21%|██        | 7/33 [00:00<00:03,  7.10it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 42ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  24%|██▍       | 8/33 [00:01<00:03,  7.20it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  27%|██▋       | 9/33 [00:01<00:03,  7.16it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  30%|███       | 10/33 [00:01<00:03,  7.09it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  33%|███▎      | 11/33 [00:01<00:03,  7.06it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  36%|███▋      | 12/33 [00:01<00:02,  7.01it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  39%|███▉      | 13/33 [00:01<00:02,  7.02it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 48ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step


Datapoints:  42%|████▏     | 14/33 [00:01<00:02,  6.96it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  45%|████▌     | 15/33 [00:02<00:02,  6.99it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 42ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  48%|████▊     | 16/33 [00:02<00:02,  7.06it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  52%|█████▏    | 17/33 [00:02<00:02,  7.12it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 48ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  55%|█████▍    | 18/33 [00:02<00:02,  6.96it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step


Datapoints:  58%|█████▊    | 19/33 [00:02<00:02,  6.93it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  61%|██████    | 20/33 [00:02<00:01,  6.88it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  64%|██████▎   | 21/33 [00:02<00:01,  6.93it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  67%|██████▋   | 22/33 [00:03<00:01,  6.91it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 46ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step


Datapoints:  70%|██████▉   | 23/33 [00:03<00:01,  6.85it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 46ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  73%|███████▎  | 24/33 [00:03<00:01,  6.81it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 50ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step


Datapoints:  76%|███████▌  | 25/33 [00:03<00:01,  6.72it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 46ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step


Datapoints:  79%|███████▉  | 26/33 [00:03<00:01,  6.71it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  82%|████████▏ | 27/33 [00:03<00:00,  6.81it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  85%|████████▍ | 28/33 [00:04<00:00,  6.86it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  88%|████████▊ | 29/33 [00:04<00:00,  6.90it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  91%|█████████ | 30/33 [00:04<00:00,  6.94it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  94%|█████████▍| 31/33 [00:04<00:00,  6.95it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  97%|█████████▋| 32/33 [00:04<00:00,  7.03it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints: 100%|██████████| 33/33 [00:04<00:00,  7.02it/s]
Datapoints: 100%|██████████| 33/33 [00:04<00:00,  6.98it/s]

CV Folds:  67%|██████▋   | 2/3 [00:18<00:09,  9.11s/it](4020, 28, 28)
Epoch 1/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 1:37 777ms/step - accuracy: 0.0938 - loss: 2.3129
 25/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.3792 - loss: 1.7809    
 49/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.4660 - loss: 1.5489
 73/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.5168 - loss: 1.4092
 97/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.5515 - loss: 1.3119
121/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.5777 - loss: 1.2377
126/126 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.5833 - loss: 1.2218
Epoch 2/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 2s 18ms/step - accuracy: 0.8125 - loss: 0.4789
 24/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7848 - loss: 0.6083 
 48/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7831 - loss: 0.6148
 72/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7856 - loss: 0.6113
 96/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7873 - loss: 0.6086
120/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7890 - loss: 0.6047
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7896 - loss: 0.6034
Epoch 3/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 3s 28ms/step - accuracy: 0.8750 - loss: 0.3564
 25/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8302 - loss: 0.5002 
 49/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8229 - loss: 0.5140
 73/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8221 - loss: 0.5145
 97/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8223 - loss: 0.5149
121/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8225 - loss: 0.5140
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8226 - loss: 0.5135
Epoch 4/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 4s 34ms/step - accuracy: 0.9062 - loss: 0.3024
 24/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8555 - loss: 0.4383 
 47/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8470 - loss: 0.4543
 70/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8456 - loss: 0.4551
 93/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8445 - loss: 0.4579
116/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8441 - loss: 0.4582
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8441 - loss: 0.4580
Epoch 5/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 2s 21ms/step - accuracy: 0.9062 - loss: 0.2785
 25/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8684 - loss: 0.4014 
 50/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8613 - loss: 0.4168
 74/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8601 - loss: 0.4188
 98/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8589 - loss: 0.4207
122/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8582 - loss: 0.4211
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8581 - loss: 0.4210
Epoch 6/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 2s 18ms/step - accuracy: 0.9062 - loss: 0.2679
 24/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8796 - loss: 0.3642 
 47/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8749 - loss: 0.3798
 70/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8738 - loss: 0.3817
 94/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8723 - loss: 0.3853
118/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8714 - loss: 0.3865
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8713 - loss: 0.3865
Epoch 7/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 2s 18ms/step - accuracy: 0.9375 - loss: 0.2306
 24/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8883 - loss: 0.3371 
 47/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8838 - loss: 0.3509
 70/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8832 - loss: 0.3527
 94/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8820 - loss: 0.3571
117/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8814 - loss: 0.3589
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8813 - loss: 0.3593
Epoch 8/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 2s 18ms/step - accuracy: 0.9375 - loss: 0.2117
 24/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8932 - loss: 0.3151 
 47/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8902 - loss: 0.3284
 70/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8893 - loss: 0.3310
 93/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8882 - loss: 0.3350
117/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8876 - loss: 0.3366
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8875 - loss: 0.3369
Epoch 9/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 3s 26ms/step - accuracy: 0.9375 - loss: 0.2051
 24/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8973 - loss: 0.2970 
 47/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8947 - loss: 0.3084
 70/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8939 - loss: 0.3106
 93/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8936 - loss: 0.3140
117/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8939 - loss: 0.3155
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8941 - loss: 0.3157
Epoch 10/10

  1/126 ━━━━━━━━━━━━━━━━━━━━ 2s 19ms/step - accuracy: 0.9375 - loss: 0.1920
 23/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8983 - loss: 0.2788 
 46/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8977 - loss: 0.2907
 68/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8982 - loss: 0.2928
 89/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8980 - loss: 0.2969
111/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8983 - loss: 0.2987
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8987 - loss: 0.2994


Datapoints:   0%|          | 0/33 [00:00<?, ?it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 46ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 25ms/step


Datapoints:   3%|▎         | 1/33 [00:00<00:04,  6.68it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:   6%|▌         | 2/33 [00:00<00:04,  6.76it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:   9%|▉         | 3/33 [00:00<00:04,  6.80it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  12%|█▏        | 4/33 [00:00<00:04,  6.83it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  15%|█▌        | 5/33 [00:00<00:04,  6.90it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  18%|█▊        | 6/33 [00:00<00:03,  6.94it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 27ms/step


Datapoints:  21%|██        | 7/33 [00:01<00:03,  6.92it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  24%|██▍       | 8/33 [00:01<00:03,  7.03it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  27%|██▋       | 9/33 [00:01<00:03,  7.11it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  30%|███       | 10/33 [00:01<00:03,  7.16it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 46ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  33%|███▎      | 11/33 [00:01<00:03,  6.98it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  36%|███▋      | 12/33 [00:01<00:03,  6.94it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step


Datapoints:  39%|███▉      | 13/33 [00:01<00:02,  6.90it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 46ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step


Datapoints:  42%|████▏     | 14/33 [00:02<00:02,  6.88it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 48ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  45%|████▌     | 15/33 [00:02<00:02,  6.74it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  48%|████▊     | 16/33 [00:02<00:02,  6.82it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 50ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  52%|█████▏    | 17/33 [00:02<00:02,  6.79it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 49ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  55%|█████▍    | 18/33 [00:02<00:02,  6.78it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  58%|█████▊    | 19/33 [00:02<00:02,  6.81it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  61%|██████    | 20/33 [00:02<00:01,  6.92it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  64%|██████▎   | 21/33 [00:03<00:01,  6.96it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  67%|██████▋   | 22/33 [00:03<00:01,  6.88it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 46ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  70%|██████▉   | 23/33 [00:03<00:01,  6.92it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 43ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  73%|███████▎  | 24/33 [00:03<00:01,  7.01it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step


Datapoints:  76%|███████▌  | 25/33 [00:03<00:01,  7.01it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints:  79%|███████▉  | 26/33 [00:03<00:01,  6.94it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step


Datapoints:  82%|████████▏ | 27/33 [00:03<00:00,  6.96it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 48ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 25ms/step


Datapoints:  85%|████████▍ | 28/33 [00:04<00:00,  6.82it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 54ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step


Datapoints:  88%|████████▊ | 29/33 [00:04<00:00,  6.61it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 47ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step


Datapoints:  91%|█████████ | 30/33 [00:04<00:00,  6.62it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step


Datapoints:  94%|█████████▍| 31/33 [00:04<00:00,  6.68it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step


Datapoints:  97%|█████████▋| 32/33 [00:04<00:00,  6.73it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 44ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step


Datapoints: 100%|██████████| 33/33 [00:04<00:00,  6.75it/s]
Datapoints: 100%|██████████| 33/33 [00:04<00:00,  6.86it/s]

CV Folds: 100%|██████████| 3/3 [00:27<00:00,  9.20s/it]
CV Folds: 100%|██████████| 3/3 [00:27<00:00,  9.17s/it]

We can now look at the results per group:

cv_results["test__single__accuracy"]

[[0.8333333333333334, 0.8333333333333334, 0.9166666666666666, 0.7666666666666667, 0.8166666666666667, 0.8166666666666667, 0.8, 0.8333333333333334, 0.8333333333333334, 0.85, 0.7666666666666667, 0.8333333333333334, 0.8833333333333333, 0.8666666666666667, 0.8833333333333333, 0.8666666666666667, 0.8333333333333334, 0.8333333333333334, 0.8, 0.8333333333333334, 0.7833333333333333, 0.7333333333333333, 0.8, 0.8333333333333334, 0.8333333333333334, 0.8833333333333333, 0.8666666666666667, 0.85, 0.85, 0.75, 0.75, 0.85, 0.8166666666666667, 0.8833333333333333], [0.8, 0.9, 0.75, 0.8666666666666667, 0.8833333333333333, 0.8166666666666667, 0.8833333333333333, 0.8833333333333333, 0.8333333333333334, 0.8, 0.85, 0.8, 0.8166666666666667, 0.8, 0.7833333333333333, 0.8, 0.8166666666666667, 0.7666666666666667, 0.85, 0.75, 0.8, 0.8333333333333334, 0.7666666666666667, 0.9, 0.8, 0.7833333333333333, 0.8166666666666667, 0.8666666666666667, 0.8833333333333333, 0.9166666666666666, 0.7333333333333333, 0.7833333333333333, 0.9], [0.7833333333333333, 0.9333333333333333, 0.75, 0.8333333333333334, 0.8833333333333333, 0.8666666666666667, 0.8833333333333333, 0.9, 0.9, 0.9, 0.9166666666666666, 0.85, 0.8666666666666667, 0.8833333333333333, 0.85, 0.8, 0.9166666666666666, 0.75, 0.7166666666666667, 0.8666666666666667, 0.8166666666666667, 0.75, 0.7833333333333333, 0.7333333333333333, 0.8666666666666667, 0.8833333333333333, 0.9166666666666666, 0.8, 0.8666666666666667, 0.9, 0.85, 0.85, 0.9]]

Average first per group and then over all groups:

cv_results["test__agg__accuracy"]

array([0.82892157, 0.82525253, 0.84747475])

And the overall accuracy as the average over all samples of all groups within a fold:

cv_results["test__agg__per_sample__accuracy"]

array([0.82892157, 0.82525253, 0.84747475])

Total running time of the script: (0 minutes 32.311 seconds)

Estimated memory usage: 247 MB

Gallery generated by Sphinx-Gallery