Tensorflow/Keras#

Note

This example requires the tensorflow package to be installed.

Theoretically, tpcp is framework agnostic and can be used with any framework. However, due to the way some frameworks handle their objects, some special handling internally is required. Hence, this example does not only serve as example on how to use tensorflow with tpcp, but also as a test case for these special cases.

When using tpcp with any machine learning framework, you either want to use a pretrained model with a normal pipeline or a train your own model as part of an Optimizable Pipeline. Here we show the second case, as it is more complex, and you are likely able to figure out the first case yourself.

This means, we are planning to perform the following steps:

  1. Create a pipeline that creates and trains a model.

  2. Allow the modification of model hyperparameters.

  3. Run a simple cross-validation to demonstrate the functionality.

This example reimplements the basic MNIST example from the [tensorflow documentation](https://www.tensorflow.org/tutorials/keras/classification).

Some Notes#

In this example we show how to implement a Pipeline that uses tensorflow. You could implement an Algorithm in a similar way. This would actually be easier, as no specific handling of the input data would be required. For a pipeline, we need to create a custom Dataset class, as this is the expected input for a pipeline.

The Dataset#

We are using the normal fashion MNIST dataset for this example It consists of 60.000 images of 28x28 pixels, each with a label. We will ignore the typical train-test split, as we want to do our own cross-validation.

In addition, we will simulate an additional “index level”. In this (and most typical deep learning datasets), each datapoint is one vector for which we can make one prediction. In tpcp, we usually deal with datasets, where you might have multiple pieces of information for each datapoint. For example, one datapoint could be a patient, for which we have an entire time series of measurements. We will simulate this here, by creating the index of our dataset as 1000 groups each containing 60 images.

Other than that, the dataset is pretty standard. Besides the create_index method, we only need to implement the input_as_array and labels_as_array methods that allow us to easily access the data once we selected a single group.

from functools import lru_cache

import numpy as np
import pandas as pd
import tensorflow as tf

from tpcp import Dataset

tf.keras.utils.set_random_seed(812)
tf.config.experimental.enable_op_determinism()


@lru_cache(maxsize=1)
def get_fashion_mnist_data():
    # Note: We throw train and test sets together, as we don't care about the official split here.
    #       We will create our own split later.
    (train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.fashion_mnist.load_data()
    return np.array(list(train_images) + list(test_images)), list(train_labels) + list(test_labels)


class FashionMNIST(Dataset):
    def input_as_array(self) -> np.ndarray:
        self.assert_is_single(None, "input_as_array")
        group_id = int(self.group_label.group_id)
        images, _ = get_fashion_mnist_data()
        return images[group_id * 60 : (group_id + 1) * 60].reshape((60, 28, 28)) / 255

    def labels_as_array(self) -> np.ndarray:
        self.assert_is_single(None, "labels_as_array")
        group_id = int(self.group_label.group_id)
        _, labels = get_fashion_mnist_data()
        return np.array(labels[group_id * 60 : (group_id + 1) * 60])

    def create_index(self) -> pd.DataFrame:
        # There are 60.000 images in total.
        # We simulate 1000 groups of 60 images each.
        return pd.DataFrame({"group_id": list(range(1000))})

We can see our Dataset works as expected:

dataset = FashionMNIST()
dataset[0].input_as_array().shape
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz

 8192/29515 [=======>......................] - ETA: 0s
29515/29515 [==============================] - 0s 1us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz

    8192/26421880 [..............................] - ETA: 0s
  122880/26421880 [..............................] - ETA: 10s
  909312/26421880 [>.............................] - ETA: 2s 
 5046272/26421880 [====>.........................] - ETA: 0s
 9904128/26421880 [==========>...................] - ETA: 0s
14852096/26421880 [===============>..............] - ETA: 0s
19505152/26421880 [=====================>........] - ETA: 0s
24387584/26421880 [==========================>...] - ETA: 0s
26421880/26421880 [==============================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz

5148/5148 [==============================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz

   8192/4422102 [..............................] - ETA: 0s
 131072/4422102 [..............................] - ETA: 1s
 671744/4422102 [===>..........................] - ETA: 0s
3997696/4422102 [==========================>...] - ETA: 0s
4422102/4422102 [==============================] - 0s 0us/step

(60, 28, 28)
dataset[0].labels_as_array().shape
(60,)

The Pipeline#

We will create a pipeline that uses a simple neural network to classify the images. In tpcp, all “things” that should be optimized need to be parameters. This means our model itself needs to be a parameter of the pipeline. However, as we don’t have the model yet, as its creation depends on other hyperparameters, we add it as an optional parameter initialized with None. Further, we prefix the parameter name with an underscore, to signify, that this is not a parameter that should be modified manually by the user. This is just convention, and it is up to you to decide how you want to name your parameters.

We further introduce a hyperparameter n_dense_layer_nodes to show how we can influence the model creation.

The optimize method#

To make our pipeline optimizable, it needs to inherit from OptimizablePipeline. Further we need to mark at least one of the parameters as OptiPara using the type annotation. We do this for our _model parameter.

Finally, we need to implement the self_optimize method. This method will get the entire training dataset as input and should update the _model parameter with the trained model. Hence, we first extract the relevant data (remember, each datapoint is 60 images), by concatinating all images over all groups in the dataset. Then we create the Keras model based on the hyperparameters. Finally, we train the model and update the _model parameter.

Here we chose to wrap the method with make_optimize_safe. This decorator will perform some runtime checks to ensure that the method is implemented correctly.

The run method#

The run method expects that the _model parameter is already set (i.e. the pipeline was already optimized). It gets a single datapoint as input (remember, a datapoint is a single group of 60 images). We then extract the data from the datapoint and let the model make a prediction. We store the prediction on our output attribute predictions_. The trailing underscore is a convention to signify, that this is an “result” attribute.

import warnings
from typing import Optional

from typing_extensions import Self

from tpcp import OptimizablePipeline, OptiPara, make_action_safe, make_optimize_safe


class KerasPipeline(OptimizablePipeline):
    n_dense_layer_nodes: int
    n_train_epochs: int
    _model: OptiPara[Optional[tf.keras.Sequential]]

    predictions_: np.ndarray

    def __init__(self, n_dense_layer_nodes=128, n_train_epochs=5, _model: Optional[tf.keras.Sequential] = None):
        self.n_dense_layer_nodes = n_dense_layer_nodes
        self.n_train_epochs = n_train_epochs
        self._model = _model

    @property
    def predicted_labels_(self):
        return np.argmax(self.predictions_, axis=1)

    @make_optimize_safe
    def self_optimize(self, dataset, **_) -> Self:
        data = np.vstack([d.input_as_array() for d in dataset])
        labels = np.hstack([d.labels_as_array() for d in dataset])

        print(data.shape)
        if self._model is not None:
            warnings.warn("Overwriting existing model!")

        self._model = tf.keras.Sequential(
            [
                tf.keras.layers.Flatten(input_shape=(28, 28)),
                tf.keras.layers.Dense(self.n_dense_layer_nodes, activation="relu"),
                tf.keras.layers.Dense(10),
            ]
        )

        self._model.compile(
            optimizer="adam",
            loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
            metrics=["accuracy"],
        )

        self._model.fit(data, labels, epochs=self.n_train_epochs)

        return self

    @make_action_safe
    def run(self, datapoint) -> Self:
        if self._model is None:
            raise RuntimeError("Model not trained yet!")
        data = datapoint.input_as_array()

        self.predictions_ = self._model.predict(data)
        return self

Testing the pipeline#

We can now test our pipeline. We will run the optimization using a couple of datapoints (to keep everything fast) and then use run to get the predictions for a single unseen datapoint.

pipeline = KerasPipeline().self_optimize(FashionMNIST()[:10])
p1 = pipeline.run(FashionMNIST()[11])
print(p1.predicted_labels_)
print(FashionMNIST()[11].labels_as_array())
(600, 28, 28)
Epoch 1/5

 1/19 [>.............................] - ETA: 19s - loss: 2.3593 - accuracy: 0.1250
13/19 [===================>..........] - ETA: 0s - loss: 1.6625 - accuracy: 0.4471 
19/19 [==============================] - 1s 5ms/step - loss: 1.5112 - accuracy: 0.4933
Epoch 2/5

 1/19 [>.............................] - ETA: 0s - loss: 0.9220 - accuracy: 0.7188
13/19 [===================>..........] - ETA: 0s - loss: 0.9038 - accuracy: 0.7115
19/19 [==============================] - 0s 5ms/step - loss: 0.8705 - accuracy: 0.7083
Epoch 3/5

 1/19 [>.............................] - ETA: 0s - loss: 1.0179 - accuracy: 0.6875
13/19 [===================>..........] - ETA: 0s - loss: 0.6895 - accuracy: 0.7764
19/19 [==============================] - 0s 5ms/step - loss: 0.6799 - accuracy: 0.7850
Epoch 4/5

 1/19 [>.............................] - ETA: 0s - loss: 0.4521 - accuracy: 0.8750
13/19 [===================>..........] - ETA: 0s - loss: 0.5492 - accuracy: 0.8197
19/19 [==============================] - 0s 5ms/step - loss: 0.5962 - accuracy: 0.8133
Epoch 5/5

 1/19 [>.............................] - ETA: 0s - loss: 0.4230 - accuracy: 0.8438
13/19 [===================>..........] - ETA: 0s - loss: 0.4875 - accuracy: 0.8510
19/19 [==============================] - 0s 4ms/step - loss: 0.5158 - accuracy: 0.8400

1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
[8 8 0 9 6 0 7 3 7 9 3 8 6 3 7 8 1 4 0 7 9 8 5 5 2 1 3 3 1 9 7 5 9 9 7 8 2
 7 2 7 2 6 7 1 1 7 5 4 8 3 5 9 0 7 3 0 0 9 1 9]
[8 8 0 9 2 0 7 3 7 9 3 8 4 3 7 8 1 4 0 7 9 8 5 5 2 1 3 4 6 7 7 5 9 9 7 8 2
 7 4 7 0 3 5 1 1 5 5 2 8 3 5 9 0 7 3 0 0 7 1 9]

We can see that even with just 5 epochs, the model already performs quite well. To quantify we can calculate the accuracy for this datapoint:

from sklearn.metrics import accuracy_score

accuracy_score(p1.predicted_labels_, FashionMNIST()[11].labels_as_array())
0.8

Cross Validation#

If we want to run a cross validation, we need to formalize the scoring into a function. We will calculate two types of accuracy: First, the accuracy per group and second, the accuracy over all images across all groups. For more information about how this works, check the Custom Scorer example.

from collections.abc import Sequence

from tpcp.validate import Aggregator


class SingleValueAccuracy(Aggregator[np.ndarray]):
    RETURN_RAW_SCORES = False

    @classmethod
    def aggregate(cls, /, values: Sequence[tuple[np.ndarray, np.ndarray]], **_) -> dict[str, float]:
        return {"accuracy": accuracy_score(np.hstack([v[0] for v in values]), np.hstack([v[1] for v in values]))}


def scoring(pipeline, datapoint):
    result: np.ndarray = pipeline.safe_run(datapoint).predicted_labels_
    reference = datapoint.labels_as_array()

    return {
        "accuracy": accuracy_score(result, reference),
        "per_sample": SingleValueAccuracy((result, reference)),
    }

Now we can run a cross validation. We will only run it on a subset of the data, to keep the runtime manageable.

Note

You might see warnings about retracing of the model. This is because we clone the pipeline before each call to the run method. This is a good idea to ensure that all pipelines are independent of each other, however, might result in some performance overhead.

from tpcp.optimize import Optimize
from tpcp.validate import cross_validate

pipeline = KerasPipeline(n_train_epochs=10)
cv_results = cross_validate(Optimize(pipeline), FashionMNIST()[:100], scoring=scoring, cv=3)
CV Folds:   0%|          | 0/3 [00:00<?, ?it/s](3960, 28, 28)
Epoch 1/10

  1/124 [..............................] - ETA: 1:55 - loss: 2.5074 - accuracy: 0.0938
 12/124 [=>............................] - ETA: 0s - loss: 1.8208 - accuracy: 0.3125  
 24/124 [====>.........................] - ETA: 0s - loss: 1.4874 - accuracy: 0.4570
 36/124 [=======>......................] - ETA: 0s - loss: 1.2993 - accuracy: 0.5278
 48/124 [==========>...................] - ETA: 0s - loss: 1.1716 - accuracy: 0.5794
 60/124 [=============>................] - ETA: 0s - loss: 1.0816 - accuracy: 0.6141
 72/124 [================>.............] - ETA: 0s - loss: 1.0366 - accuracy: 0.6319
 84/124 [===================>..........] - ETA: 0s - loss: 0.9932 - accuracy: 0.6484
 96/124 [======================>.......] - ETA: 0s - loss: 0.9648 - accuracy: 0.6621
108/124 [=========================>....] - ETA: 0s - loss: 0.9299 - accuracy: 0.6756
120/124 [============================>.] - ETA: 0s - loss: 0.9033 - accuracy: 0.6883
124/124 [==============================] - 1s 5ms/step - loss: 0.8968 - accuracy: 0.6919
Epoch 2/10

  1/124 [..............................] - ETA: 0s - loss: 1.0481 - accuracy: 0.7500
 13/124 [==>...........................] - ETA: 0s - loss: 0.6082 - accuracy: 0.8005
 24/124 [====>.........................] - ETA: 0s - loss: 0.5817 - accuracy: 0.8112
 36/124 [=======>......................] - ETA: 0s - loss: 0.6109 - accuracy: 0.7925
 48/124 [==========>...................] - ETA: 0s - loss: 0.5921 - accuracy: 0.8008
 61/124 [=============>................] - ETA: 0s - loss: 0.5840 - accuracy: 0.7976
 73/124 [================>.............] - ETA: 0s - loss: 0.5668 - accuracy: 0.8074
 85/124 [===================>..........] - ETA: 0s - loss: 0.5685 - accuracy: 0.8062
 97/124 [======================>.......] - ETA: 0s - loss: 0.5629 - accuracy: 0.8115
109/124 [=========================>....] - ETA: 0s - loss: 0.5591 - accuracy: 0.8114
121/124 [============================>.] - ETA: 0s - loss: 0.5576 - accuracy: 0.8122
124/124 [==============================] - 1s 4ms/step - loss: 0.5606 - accuracy: 0.8104
Epoch 3/10

  1/124 [..............................] - ETA: 0s - loss: 0.7180 - accuracy: 0.6250
 13/124 [==>...........................] - ETA: 0s - loss: 0.5132 - accuracy: 0.8053
 25/124 [=====>........................] - ETA: 0s - loss: 0.5200 - accuracy: 0.8125
 37/124 [=======>......................] - ETA: 0s - loss: 0.5243 - accuracy: 0.8218
 49/124 [==========>...................] - ETA: 0s - loss: 0.5135 - accuracy: 0.8233
 61/124 [=============>................] - ETA: 0s - loss: 0.5053 - accuracy: 0.8222
 73/124 [================>.............] - ETA: 0s - loss: 0.5060 - accuracy: 0.8215
 85/124 [===================>..........] - ETA: 0s - loss: 0.4954 - accuracy: 0.8257
 97/124 [======================>.......] - ETA: 0s - loss: 0.4940 - accuracy: 0.8264
109/124 [=========================>....] - ETA: 0s - loss: 0.4974 - accuracy: 0.8243
122/124 [============================>.] - ETA: 0s - loss: 0.4976 - accuracy: 0.8240
124/124 [==============================] - 1s 4ms/step - loss: 0.5009 - accuracy: 0.8232
Epoch 4/10

  1/124 [..............................] - ETA: 0s - loss: 0.4630 - accuracy: 0.8125
 13/124 [==>...........................] - ETA: 0s - loss: 0.3962 - accuracy: 0.8630
 25/124 [=====>........................] - ETA: 0s - loss: 0.4104 - accuracy: 0.8587
 37/124 [=======>......................] - ETA: 0s - loss: 0.4127 - accuracy: 0.8556
 48/124 [==========>...................] - ETA: 0s - loss: 0.4300 - accuracy: 0.8522
 58/124 [=============>................] - ETA: 0s - loss: 0.4337 - accuracy: 0.8491
 69/124 [===============>..............] - ETA: 0s - loss: 0.4418 - accuracy: 0.8478
 81/124 [==================>...........] - ETA: 0s - loss: 0.4444 - accuracy: 0.8484
 93/124 [=====================>........] - ETA: 0s - loss: 0.4434 - accuracy: 0.8464
105/124 [========================>.....] - ETA: 0s - loss: 0.4405 - accuracy: 0.8467
117/124 [===========================>..] - ETA: 0s - loss: 0.4450 - accuracy: 0.8438
124/124 [==============================] - 1s 5ms/step - loss: 0.4448 - accuracy: 0.8437
Epoch 5/10

  1/124 [..............................] - ETA: 0s - loss: 0.6924 - accuracy: 0.6250
 13/124 [==>...........................] - ETA: 0s - loss: 0.4504 - accuracy: 0.8173
 25/124 [=====>........................] - ETA: 0s - loss: 0.4423 - accuracy: 0.8263
 37/124 [=======>......................] - ETA: 0s - loss: 0.4455 - accuracy: 0.8294
 49/124 [==========>...................] - ETA: 0s - loss: 0.4360 - accuracy: 0.8399
 61/124 [=============>................] - ETA: 0s - loss: 0.4225 - accuracy: 0.8438
 73/124 [================>.............] - ETA: 0s - loss: 0.4221 - accuracy: 0.8485
 85/124 [===================>..........] - ETA: 0s - loss: 0.4100 - accuracy: 0.8540
 97/124 [======================>.......] - ETA: 0s - loss: 0.4067 - accuracy: 0.8560
109/124 [=========================>....] - ETA: 0s - loss: 0.4122 - accuracy: 0.8544
121/124 [============================>.] - ETA: 0s - loss: 0.4119 - accuracy: 0.8546
124/124 [==============================] - 1s 4ms/step - loss: 0.4119 - accuracy: 0.8540
Epoch 6/10

  1/124 [..............................] - ETA: 0s - loss: 0.5976 - accuracy: 0.7812
 14/124 [==>...........................] - ETA: 0s - loss: 0.3864 - accuracy: 0.8705
 26/124 [=====>........................] - ETA: 0s - loss: 0.4092 - accuracy: 0.8558
 38/124 [========>.....................] - ETA: 0s - loss: 0.3782 - accuracy: 0.8660
 50/124 [===========>..................] - ETA: 0s - loss: 0.3883 - accuracy: 0.8650
 62/124 [==============>...............] - ETA: 0s - loss: 0.3825 - accuracy: 0.8669
 74/124 [================>.............] - ETA: 0s - loss: 0.3824 - accuracy: 0.8657
 86/124 [===================>..........] - ETA: 0s - loss: 0.3877 - accuracy: 0.8645
 98/124 [======================>.......] - ETA: 0s - loss: 0.3790 - accuracy: 0.8686
110/124 [=========================>....] - ETA: 0s - loss: 0.3823 - accuracy: 0.8673
122/124 [============================>.] - ETA: 0s - loss: 0.3810 - accuracy: 0.8678
124/124 [==============================] - 1s 4ms/step - loss: 0.3792 - accuracy: 0.8687
Epoch 7/10

  1/124 [..............................] - ETA: 0s - loss: 0.4009 - accuracy: 0.8438
 13/124 [==>...........................] - ETA: 0s - loss: 0.3694 - accuracy: 0.8726
 25/124 [=====>........................] - ETA: 0s - loss: 0.3564 - accuracy: 0.8700
 37/124 [=======>......................] - ETA: 0s - loss: 0.3387 - accuracy: 0.8758
 49/124 [==========>...................] - ETA: 0s - loss: 0.3485 - accuracy: 0.8744
 61/124 [=============>................] - ETA: 0s - loss: 0.3571 - accuracy: 0.8760
 73/124 [================>.............] - ETA: 0s - loss: 0.3593 - accuracy: 0.8750
 85/124 [===================>..........] - ETA: 0s - loss: 0.3578 - accuracy: 0.8757
 97/124 [======================>.......] - ETA: 0s - loss: 0.3618 - accuracy: 0.8766
109/124 [=========================>....] - ETA: 0s - loss: 0.3580 - accuracy: 0.8767
121/124 [============================>.] - ETA: 0s - loss: 0.3582 - accuracy: 0.8755
124/124 [==============================] - 1s 4ms/step - loss: 0.3555 - accuracy: 0.8768
Epoch 8/10

  1/124 [..............................] - ETA: 0s - loss: 0.2810 - accuracy: 0.9062
 13/124 [==>...........................] - ETA: 0s - loss: 0.3236 - accuracy: 0.8966
 25/124 [=====>........................] - ETA: 0s - loss: 0.2929 - accuracy: 0.9150
 37/124 [=======>......................] - ETA: 0s - loss: 0.2969 - accuracy: 0.9079
 49/124 [==========>...................] - ETA: 0s - loss: 0.3031 - accuracy: 0.9024
 61/124 [=============>................] - ETA: 0s - loss: 0.3088 - accuracy: 0.8981
 73/124 [================>.............] - ETA: 0s - loss: 0.3173 - accuracy: 0.8943
 85/124 [===================>..........] - ETA: 0s - loss: 0.3251 - accuracy: 0.8926
 97/124 [======================>.......] - ETA: 0s - loss: 0.3274 - accuracy: 0.8914
109/124 [=========================>....] - ETA: 0s - loss: 0.3369 - accuracy: 0.8879
121/124 [============================>.] - ETA: 0s - loss: 0.3466 - accuracy: 0.8825
124/124 [==============================] - 1s 4ms/step - loss: 0.3464 - accuracy: 0.8826
Epoch 9/10

  1/124 [..............................] - ETA: 0s - loss: 0.2351 - accuracy: 0.9375
 13/124 [==>...........................] - ETA: 0s - loss: 0.3713 - accuracy: 0.8726
 25/124 [=====>........................] - ETA: 0s - loss: 0.3446 - accuracy: 0.8850
 37/124 [=======>......................] - ETA: 0s - loss: 0.3441 - accuracy: 0.8742
 49/124 [==========>...................] - ETA: 0s - loss: 0.3385 - accuracy: 0.8801
 60/124 [=============>................] - ETA: 0s - loss: 0.3325 - accuracy: 0.8828
 70/124 [===============>..............] - ETA: 0s - loss: 0.3258 - accuracy: 0.8862
 82/124 [==================>...........] - ETA: 0s - loss: 0.3134 - accuracy: 0.8921
 94/124 [=====================>........] - ETA: 0s - loss: 0.3241 - accuracy: 0.8893
106/124 [========================>.....] - ETA: 0s - loss: 0.3215 - accuracy: 0.8912
118/124 [===========================>..] - ETA: 0s - loss: 0.3186 - accuracy: 0.8893
124/124 [==============================] - 1s 5ms/step - loss: 0.3196 - accuracy: 0.8886
Epoch 10/10

  1/124 [..............................] - ETA: 0s - loss: 0.2527 - accuracy: 0.9688
 13/124 [==>...........................] - ETA: 0s - loss: 0.2215 - accuracy: 0.9255
 25/124 [=====>........................] - ETA: 0s - loss: 0.2457 - accuracy: 0.9125
 37/124 [=======>......................] - ETA: 0s - loss: 0.2692 - accuracy: 0.9037
 49/124 [==========>...................] - ETA: 0s - loss: 0.2755 - accuracy: 0.9031
 61/124 [=============>................] - ETA: 0s - loss: 0.2910 - accuracy: 0.9001
 73/124 [================>.............] - ETA: 0s - loss: 0.2991 - accuracy: 0.8951
 85/124 [===================>..........] - ETA: 0s - loss: 0.2970 - accuracy: 0.8960
 97/124 [======================>.......] - ETA: 0s - loss: 0.2953 - accuracy: 0.8982
109/124 [=========================>....] - ETA: 0s - loss: 0.2941 - accuracy: 0.8999
121/124 [============================>.] - ETA: 0s - loss: 0.2929 - accuracy: 0.8993
124/124 [==============================] - 1s 4ms/step - loss: 0.2921 - accuracy: 0.8997


Datapoints:   0%|          | 0/34 [00:00<?, ?it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:   3%|▎         | 1/34 [00:00<00:09,  3.54it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:   6%|▌         | 2/34 [00:00<00:09,  3.46it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:   9%|▉         | 3/34 [00:00<00:08,  3.47it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  12%|█▏        | 4/34 [00:01<00:08,  3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  15%|█▍        | 5/34 [00:01<00:08,  3.48it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  18%|█▊        | 6/34 [00:01<00:08,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  21%|██        | 7/34 [00:02<00:07,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  24%|██▎       | 8/34 [00:02<00:07,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  26%|██▋       | 9/34 [00:02<00:07,  3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  29%|██▉       | 10/34 [00:02<00:07,  3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  32%|███▏      | 11/34 [00:03<00:06,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  35%|███▌      | 12/34 [00:03<00:06,  3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  38%|███▊      | 13/34 [00:03<00:06,  3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  41%|████      | 14/34 [00:04<00:05,  3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  44%|████▍     | 15/34 [00:04<00:05,  3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  47%|████▋     | 16/34 [00:04<00:05,  3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  50%|█████     | 17/34 [00:04<00:04,  3.47it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  53%|█████▎    | 18/34 [00:05<00:04,  3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  56%|█████▌    | 19/34 [00:05<00:04,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  59%|█████▉    | 20/34 [00:05<00:04,  3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  62%|██████▏   | 21/34 [00:06<00:03,  3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  65%|██████▍   | 22/34 [00:06<00:03,  3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  68%|██████▊   | 23/34 [00:06<00:03,  3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  71%|███████   | 24/34 [00:07<00:02,  3.37it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  74%|███████▎  | 25/34 [00:07<00:02,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  76%|███████▋  | 26/34 [00:07<00:02,  3.47it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  79%|███████▉  | 27/34 [00:07<00:02,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  82%|████████▏ | 28/34 [00:08<00:01,  3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  85%|████████▌ | 29/34 [00:08<00:01,  3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  88%|████████▊ | 30/34 [00:08<00:01,  3.37it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  91%|█████████ | 31/34 [00:09<00:00,  3.37it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  94%|█████████▍| 32/34 [00:09<00:00,  3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  97%|█████████▋| 33/34 [00:09<00:00,  3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints: 100%|██████████| 34/34 [00:09<00:00,  3.39it/s]
Datapoints: 100%|██████████| 34/34 [00:09<00:00,  3.41it/s]

CV Folds:  33%|███▎      | 1/3 [00:22<00:45, 22.57s/it](4020, 28, 28)
Epoch 1/10

  1/126 [..............................] - ETA: 1:55 - loss: 2.3656 - accuracy: 0.1250
 13/126 [==>...........................] - ETA: 0s - loss: 1.7826 - accuracy: 0.3870  
 25/126 [====>.........................] - ETA: 0s - loss: 1.4578 - accuracy: 0.5150
 37/126 [=======>......................] - ETA: 0s - loss: 1.3063 - accuracy: 0.5633
 49/126 [==========>...................] - ETA: 0s - loss: 1.2005 - accuracy: 0.6014
 61/126 [=============>................] - ETA: 0s - loss: 1.1294 - accuracy: 0.6168
 73/126 [================>.............] - ETA: 0s - loss: 1.0706 - accuracy: 0.6378
 85/126 [===================>..........] - ETA: 0s - loss: 1.0088 - accuracy: 0.6603
 97/126 [======================>.......] - ETA: 0s - loss: 0.9718 - accuracy: 0.6727
109/126 [========================>.....] - ETA: 0s - loss: 0.9331 - accuracy: 0.6841
121/126 [===========================>..] - ETA: 0s - loss: 0.9028 - accuracy: 0.6955
126/126 [==============================] - 1s 4ms/step - loss: 0.8913 - accuracy: 0.7000
Epoch 2/10

  1/126 [..............................] - ETA: 0s - loss: 0.4984 - accuracy: 0.8125
 13/126 [==>...........................] - ETA: 0s - loss: 0.5529 - accuracy: 0.7957
 25/126 [====>.........................] - ETA: 0s - loss: 0.5682 - accuracy: 0.8037
 37/126 [=======>......................] - ETA: 0s - loss: 0.5658 - accuracy: 0.7990
 49/126 [==========>...................] - ETA: 0s - loss: 0.5778 - accuracy: 0.7927
 61/126 [=============>................] - ETA: 0s - loss: 0.5962 - accuracy: 0.7874
 73/126 [================>.............] - ETA: 0s - loss: 0.5902 - accuracy: 0.7902
 85/126 [===================>..........] - ETA: 0s - loss: 0.5773 - accuracy: 0.7926
 97/126 [======================>.......] - ETA: 0s - loss: 0.5772 - accuracy: 0.7938
109/126 [========================>.....] - ETA: 0s - loss: 0.5713 - accuracy: 0.7967
121/126 [===========================>..] - ETA: 0s - loss: 0.5649 - accuracy: 0.7998
126/126 [==============================] - 1s 4ms/step - loss: 0.5662 - accuracy: 0.8002
Epoch 3/10

  1/126 [..............................] - ETA: 0s - loss: 0.5249 - accuracy: 0.8125
 13/126 [==>...........................] - ETA: 0s - loss: 0.5380 - accuracy: 0.8125
 25/126 [====>.........................] - ETA: 0s - loss: 0.5201 - accuracy: 0.8275
 37/126 [=======>......................] - ETA: 0s - loss: 0.4840 - accuracy: 0.8353
 49/126 [==========>...................] - ETA: 0s - loss: 0.4717 - accuracy: 0.8329
 61/126 [=============>................] - ETA: 0s - loss: 0.4678 - accuracy: 0.8381
 73/126 [================>.............] - ETA: 0s - loss: 0.4692 - accuracy: 0.8373
 85/126 [===================>..........] - ETA: 0s - loss: 0.4810 - accuracy: 0.8342
 97/126 [======================>.......] - ETA: 0s - loss: 0.4782 - accuracy: 0.8354
109/126 [========================>.....] - ETA: 0s - loss: 0.4805 - accuracy: 0.8343
121/126 [===========================>..] - ETA: 0s - loss: 0.4901 - accuracy: 0.8275
126/126 [==============================] - 1s 4ms/step - loss: 0.4893 - accuracy: 0.8279
Epoch 4/10

  1/126 [..............................] - ETA: 0s - loss: 0.2082 - accuracy: 0.9688
 13/126 [==>...........................] - ETA: 0s - loss: 0.4042 - accuracy: 0.8654
 25/126 [====>.........................] - ETA: 0s - loss: 0.4238 - accuracy: 0.8587
 37/126 [=======>......................] - ETA: 0s - loss: 0.4437 - accuracy: 0.8480
 49/126 [==========>...................] - ETA: 0s - loss: 0.4393 - accuracy: 0.8469
 61/126 [=============>................] - ETA: 0s - loss: 0.4340 - accuracy: 0.8478
 73/126 [================>.............] - ETA: 0s - loss: 0.4283 - accuracy: 0.8527
 85/126 [===================>..........] - ETA: 0s - loss: 0.4348 - accuracy: 0.8526
 97/126 [======================>.......] - ETA: 0s - loss: 0.4394 - accuracy: 0.8502
109/126 [========================>.....] - ETA: 0s - loss: 0.4472 - accuracy: 0.8438
121/126 [===========================>..] - ETA: 0s - loss: 0.4457 - accuracy: 0.8450
126/126 [==============================] - 1s 4ms/step - loss: 0.4431 - accuracy: 0.8453
Epoch 5/10

  1/126 [..............................] - ETA: 0s - loss: 0.1979 - accuracy: 0.9375
 13/126 [==>...........................] - ETA: 0s - loss: 0.3542 - accuracy: 0.8822
 25/126 [====>.........................] - ETA: 0s - loss: 0.3946 - accuracy: 0.8612
 37/126 [=======>......................] - ETA: 0s - loss: 0.4174 - accuracy: 0.8471
 49/126 [==========>...................] - ETA: 0s - loss: 0.4057 - accuracy: 0.8540
 61/126 [=============>................] - ETA: 0s - loss: 0.4065 - accuracy: 0.8535
 73/126 [================>.............] - ETA: 0s - loss: 0.4017 - accuracy: 0.8570
 85/126 [===================>..........] - ETA: 0s - loss: 0.4044 - accuracy: 0.8570
 97/126 [======================>.......] - ETA: 0s - loss: 0.4087 - accuracy: 0.8544
109/126 [========================>.....] - ETA: 0s - loss: 0.4048 - accuracy: 0.8569
121/126 [===========================>..] - ETA: 0s - loss: 0.4056 - accuracy: 0.8574
126/126 [==============================] - 1s 4ms/step - loss: 0.4046 - accuracy: 0.8580
Epoch 6/10

  1/126 [..............................] - ETA: 0s - loss: 0.4598 - accuracy: 0.8125
 13/126 [==>...........................] - ETA: 0s - loss: 0.3434 - accuracy: 0.8534
 25/126 [====>.........................] - ETA: 0s - loss: 0.3639 - accuracy: 0.8625
 37/126 [=======>......................] - ETA: 0s - loss: 0.3791 - accuracy: 0.8573
 49/126 [==========>...................] - ETA: 0s - loss: 0.3696 - accuracy: 0.8642
 61/126 [=============>................] - ETA: 0s - loss: 0.3709 - accuracy: 0.8622
 73/126 [================>.............] - ETA: 0s - loss: 0.3741 - accuracy: 0.8647
 85/126 [===================>..........] - ETA: 0s - loss: 0.3770 - accuracy: 0.8654
 97/126 [======================>.......] - ETA: 0s - loss: 0.3796 - accuracy: 0.8650
109/126 [========================>.....] - ETA: 0s - loss: 0.3698 - accuracy: 0.8687
121/126 [===========================>..] - ETA: 0s - loss: 0.3775 - accuracy: 0.8662
126/126 [==============================] - 1s 4ms/step - loss: 0.3772 - accuracy: 0.8667
Epoch 7/10

  1/126 [..............................] - ETA: 0s - loss: 0.2244 - accuracy: 0.9062
 13/126 [==>...........................] - ETA: 0s - loss: 0.3541 - accuracy: 0.8798
 25/126 [====>.........................] - ETA: 0s - loss: 0.3532 - accuracy: 0.8813
 37/126 [=======>......................] - ETA: 0s - loss: 0.3349 - accuracy: 0.8843
 49/126 [==========>...................] - ETA: 0s - loss: 0.3264 - accuracy: 0.8884
 61/126 [=============>................] - ETA: 0s - loss: 0.3308 - accuracy: 0.8858
 73/126 [================>.............] - ETA: 0s - loss: 0.3448 - accuracy: 0.8823
 85/126 [===================>..........] - ETA: 0s - loss: 0.3522 - accuracy: 0.8790
 97/126 [======================>.......] - ETA: 0s - loss: 0.3469 - accuracy: 0.8798
109/126 [========================>.....] - ETA: 0s - loss: 0.3480 - accuracy: 0.8802
121/126 [===========================>..] - ETA: 0s - loss: 0.3487 - accuracy: 0.8786
126/126 [==============================] - 1s 4ms/step - loss: 0.3479 - accuracy: 0.8786
Epoch 8/10

  1/126 [..............................] - ETA: 0s - loss: 0.2092 - accuracy: 0.9375
 13/126 [==>...........................] - ETA: 0s - loss: 0.3092 - accuracy: 0.8942
 25/126 [====>.........................] - ETA: 0s - loss: 0.2997 - accuracy: 0.9000
 37/126 [=======>......................] - ETA: 0s - loss: 0.2936 - accuracy: 0.8986
 49/126 [==========>...................] - ETA: 0s - loss: 0.3115 - accuracy: 0.8935
 61/126 [=============>................] - ETA: 0s - loss: 0.3090 - accuracy: 0.8934
 73/126 [================>.............] - ETA: 0s - loss: 0.3184 - accuracy: 0.8887
 85/126 [===================>..........] - ETA: 0s - loss: 0.3252 - accuracy: 0.8875
 97/126 [======================>.......] - ETA: 0s - loss: 0.3264 - accuracy: 0.8869
109/126 [========================>.....] - ETA: 0s - loss: 0.3268 - accuracy: 0.8885
121/126 [===========================>..] - ETA: 0s - loss: 0.3287 - accuracy: 0.8861
126/126 [==============================] - 1s 4ms/step - loss: 0.3285 - accuracy: 0.8863
Epoch 9/10

  1/126 [..............................] - ETA: 0s - loss: 0.1886 - accuracy: 0.9062
 13/126 [==>...........................] - ETA: 0s - loss: 0.2965 - accuracy: 0.9038
 25/126 [====>.........................] - ETA: 0s - loss: 0.2998 - accuracy: 0.9000
 37/126 [=======>......................] - ETA: 0s - loss: 0.2976 - accuracy: 0.8953
 49/126 [==========>...................] - ETA: 0s - loss: 0.3057 - accuracy: 0.8909
 61/126 [=============>................] - ETA: 0s - loss: 0.3055 - accuracy: 0.8842
 73/126 [================>.............] - ETA: 0s - loss: 0.3189 - accuracy: 0.8789
 85/126 [===================>..........] - ETA: 0s - loss: 0.3181 - accuracy: 0.8809
 97/126 [======================>.......] - ETA: 0s - loss: 0.3062 - accuracy: 0.8860
109/126 [========================>.....] - ETA: 0s - loss: 0.3139 - accuracy: 0.8862
121/126 [===========================>..] - ETA: 0s - loss: 0.3122 - accuracy: 0.8892
126/126 [==============================] - 1s 4ms/step - loss: 0.3120 - accuracy: 0.8900
Epoch 10/10

  1/126 [..............................] - ETA: 0s - loss: 0.2327 - accuracy: 0.9062
 13/126 [==>...........................] - ETA: 0s - loss: 0.2908 - accuracy: 0.8870
 25/126 [====>.........................] - ETA: 0s - loss: 0.2960 - accuracy: 0.8825
 37/126 [=======>......................] - ETA: 0s - loss: 0.3081 - accuracy: 0.8851
 49/126 [==========>...................] - ETA: 0s - loss: 0.2985 - accuracy: 0.8903
 61/126 [=============>................] - ETA: 0s - loss: 0.2978 - accuracy: 0.8888
 73/126 [================>.............] - ETA: 0s - loss: 0.3040 - accuracy: 0.8883
 85/126 [===================>..........] - ETA: 0s - loss: 0.3004 - accuracy: 0.8886
 97/126 [======================>.......] - ETA: 0s - loss: 0.3038 - accuracy: 0.8876
109/126 [========================>.....] - ETA: 0s - loss: 0.3052 - accuracy: 0.8882
121/126 [===========================>..] - ETA: 0s - loss: 0.3039 - accuracy: 0.8882
126/126 [==============================] - 1s 4ms/step - loss: 0.3065 - accuracy: 0.8871


Datapoints:   0%|          | 0/33 [00:00<?, ?it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:   3%|▎         | 1/33 [00:00<00:08,  3.58it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:   6%|▌         | 2/33 [00:00<00:08,  3.48it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:   9%|▉         | 3/33 [00:00<00:08,  3.45it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  12%|█▏        | 4/33 [00:01<00:08,  3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  15%|█▌        | 5/33 [00:01<00:08,  3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  18%|█▊        | 6/33 [00:01<00:07,  3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  21%|██        | 7/33 [00:02<00:07,  3.45it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  24%|██▍       | 8/33 [00:02<00:07,  3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  27%|██▋       | 9/33 [00:02<00:07,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  30%|███       | 10/33 [00:02<00:06,  3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  33%|███▎      | 11/33 [00:03<00:06,  3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  36%|███▋      | 12/33 [00:03<00:06,  3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  39%|███▉      | 13/33 [00:03<00:05,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  42%|████▏     | 14/33 [00:04<00:05,  3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  45%|████▌     | 15/33 [00:04<00:05,  3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  48%|████▊     | 16/33 [00:04<00:05,  3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  52%|█████▏    | 17/33 [00:04<00:04,  3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  55%|█████▍    | 18/33 [00:05<00:04,  3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  58%|█████▊    | 19/33 [00:05<00:04,  3.45it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  61%|██████    | 20/33 [00:05<00:03,  3.48it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  64%|██████▎   | 21/33 [00:06<00:03,  3.52it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  67%|██████▋   | 22/33 [00:06<00:03,  3.45it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  70%|██████▉   | 23/33 [00:06<00:02,  3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  73%|███████▎  | 24/33 [00:06<00:02,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  76%|███████▌  | 25/33 [00:07<00:02,  3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  79%|███████▉  | 26/33 [00:07<00:02,  3.48it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  82%|████████▏ | 27/33 [00:07<00:01,  3.51it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  85%|████████▍ | 28/33 [00:08<00:01,  3.45it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  88%|████████▊ | 29/33 [00:08<00:01,  3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  91%|█████████ | 30/33 [00:08<00:00,  3.48it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  94%|█████████▍| 31/33 [00:09<00:00,  3.48it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  97%|█████████▋| 32/33 [00:09<00:00,  3.51it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints: 100%|██████████| 33/33 [00:09<00:00,  3.52it/s]
Datapoints: 100%|██████████| 33/33 [00:09<00:00,  3.45it/s]

CV Folds:  67%|██████▋   | 2/3 [00:44<00:22, 22.31s/it](4020, 28, 28)
Epoch 1/10

  1/126 [..............................] - ETA: 1:56 - loss: 2.3129 - accuracy: 0.0938
 12/126 [=>............................] - ETA: 0s - loss: 1.7505 - accuracy: 0.3984  
 24/126 [====>.........................] - ETA: 0s - loss: 1.4718 - accuracy: 0.5026
 36/126 [=======>......................] - ETA: 0s - loss: 1.3248 - accuracy: 0.5460
 48/126 [==========>...................] - ETA: 0s - loss: 1.2118 - accuracy: 0.5905
 60/126 [=============>................] - ETA: 0s - loss: 1.1265 - accuracy: 0.6182
 72/126 [================>.............] - ETA: 0s - loss: 1.0826 - accuracy: 0.6328
 84/126 [===================>..........] - ETA: 0s - loss: 1.0169 - accuracy: 0.6562
 96/126 [=====================>........] - ETA: 0s - loss: 0.9770 - accuracy: 0.6712
108/126 [========================>.....] - ETA: 0s - loss: 0.9428 - accuracy: 0.6808
120/126 [===========================>..] - ETA: 0s - loss: 0.9144 - accuracy: 0.6922
126/126 [==============================] - 1s 4ms/step - loss: 0.8965 - accuracy: 0.6983
Epoch 2/10

  1/126 [..............................] - ETA: 0s - loss: 0.4769 - accuracy: 0.8125
 13/126 [==>...........................] - ETA: 0s - loss: 0.5761 - accuracy: 0.7957
 25/126 [====>.........................] - ETA: 0s - loss: 0.6120 - accuracy: 0.7912
 37/126 [=======>......................] - ETA: 0s - loss: 0.6063 - accuracy: 0.7889
 49/126 [==========>...................] - ETA: 0s - loss: 0.5931 - accuracy: 0.7972
 61/126 [=============>................] - ETA: 0s - loss: 0.5958 - accuracy: 0.7935
 73/126 [================>.............] - ETA: 0s - loss: 0.5951 - accuracy: 0.7954
 85/126 [===================>..........] - ETA: 0s - loss: 0.5894 - accuracy: 0.7967
 97/126 [======================>.......] - ETA: 0s - loss: 0.5817 - accuracy: 0.8015
109/126 [========================>.....] - ETA: 0s - loss: 0.5710 - accuracy: 0.8048
121/126 [===========================>..] - ETA: 0s - loss: 0.5654 - accuracy: 0.8068
126/126 [==============================] - 1s 4ms/step - loss: 0.5750 - accuracy: 0.8040
Epoch 3/10

  1/126 [..............................] - ETA: 0s - loss: 0.6893 - accuracy: 0.7812
 13/126 [==>...........................] - ETA: 0s - loss: 0.5276 - accuracy: 0.8245
 25/126 [====>.........................] - ETA: 0s - loss: 0.5681 - accuracy: 0.8075
 37/126 [=======>......................] - ETA: 0s - loss: 0.5517 - accuracy: 0.8083
 49/126 [==========>...................] - ETA: 0s - loss: 0.5377 - accuracy: 0.8119
 61/126 [=============>................] - ETA: 0s - loss: 0.5310 - accuracy: 0.8166
 73/126 [================>.............] - ETA: 0s - loss: 0.5269 - accuracy: 0.8185
 85/126 [===================>..........] - ETA: 0s - loss: 0.5284 - accuracy: 0.8210
 97/126 [======================>.......] - ETA: 0s - loss: 0.5211 - accuracy: 0.8212
109/126 [========================>.....] - ETA: 0s - loss: 0.5114 - accuracy: 0.8248
121/126 [===========================>..] - ETA: 0s - loss: 0.5134 - accuracy: 0.8236
126/126 [==============================] - 1s 4ms/step - loss: 0.5140 - accuracy: 0.8224
Epoch 4/10

  1/126 [..............................] - ETA: 0s - loss: 0.4519 - accuracy: 0.9062
 13/126 [==>...........................] - ETA: 0s - loss: 0.4019 - accuracy: 0.8750
 25/126 [====>.........................] - ETA: 0s - loss: 0.4193 - accuracy: 0.8587
 37/126 [=======>......................] - ETA: 0s - loss: 0.4405 - accuracy: 0.8480
 49/126 [==========>...................] - ETA: 0s - loss: 0.4532 - accuracy: 0.8399
 61/126 [=============>................] - ETA: 0s - loss: 0.4511 - accuracy: 0.8412
 73/126 [================>.............] - ETA: 0s - loss: 0.4450 - accuracy: 0.8438
 85/126 [===================>..........] - ETA: 0s - loss: 0.4464 - accuracy: 0.8438
 97/126 [======================>.......] - ETA: 0s - loss: 0.4548 - accuracy: 0.8415
109/126 [========================>.....] - ETA: 0s - loss: 0.4619 - accuracy: 0.8394
121/126 [===========================>..] - ETA: 0s - loss: 0.4629 - accuracy: 0.8383
126/126 [==============================] - 1s 4ms/step - loss: 0.4667 - accuracy: 0.8358
Epoch 5/10

  1/126 [..............................] - ETA: 0s - loss: 0.4192 - accuracy: 0.8125
 13/126 [==>...........................] - ETA: 0s - loss: 0.4263 - accuracy: 0.8654
 25/126 [====>.........................] - ETA: 0s - loss: 0.4182 - accuracy: 0.8625
 37/126 [=======>......................] - ETA: 0s - loss: 0.4131 - accuracy: 0.8556
 49/126 [==========>...................] - ETA: 0s - loss: 0.4216 - accuracy: 0.8635
 61/126 [=============>................] - ETA: 0s - loss: 0.4160 - accuracy: 0.8648
 73/126 [================>.............] - ETA: 0s - loss: 0.4160 - accuracy: 0.8634
 85/126 [===================>..........] - ETA: 0s - loss: 0.4222 - accuracy: 0.8599
 97/126 [======================>.......] - ETA: 0s - loss: 0.4279 - accuracy: 0.8595
109/126 [========================>.....] - ETA: 0s - loss: 0.4202 - accuracy: 0.8595
121/126 [===========================>..] - ETA: 0s - loss: 0.4177 - accuracy: 0.8598
126/126 [==============================] - 1s 4ms/step - loss: 0.4173 - accuracy: 0.8600
Epoch 6/10

  1/126 [..............................] - ETA: 0s - loss: 0.8799 - accuracy: 0.7500
 13/126 [==>...........................] - ETA: 0s - loss: 0.3963 - accuracy: 0.8582
 25/126 [====>.........................] - ETA: 0s - loss: 0.3716 - accuracy: 0.8662
 37/126 [=======>......................] - ETA: 0s - loss: 0.3773 - accuracy: 0.8674
 49/126 [==========>...................] - ETA: 0s - loss: 0.3820 - accuracy: 0.8699
 60/126 [=============>................] - ETA: 0s - loss: 0.3773 - accuracy: 0.8672
 72/126 [================>.............] - ETA: 0s - loss: 0.3877 - accuracy: 0.8672
 84/126 [===================>..........] - ETA: 0s - loss: 0.3925 - accuracy: 0.8650
 96/126 [=====================>........] - ETA: 0s - loss: 0.3944 - accuracy: 0.8620
108/126 [========================>.....] - ETA: 0s - loss: 0.3934 - accuracy: 0.8623
120/126 [===========================>..] - ETA: 0s - loss: 0.3980 - accuracy: 0.8628
126/126 [==============================] - 1s 4ms/step - loss: 0.3949 - accuracy: 0.8624
Epoch 7/10

  1/126 [..............................] - ETA: 0s - loss: 0.2022 - accuracy: 0.9375
 13/126 [==>...........................] - ETA: 0s - loss: 0.3838 - accuracy: 0.8774
 25/126 [====>.........................] - ETA: 0s - loss: 0.3768 - accuracy: 0.8775
 37/126 [=======>......................] - ETA: 0s - loss: 0.3513 - accuracy: 0.8801
 49/126 [==========>...................] - ETA: 0s - loss: 0.3411 - accuracy: 0.8852
 61/126 [=============>................] - ETA: 0s - loss: 0.3486 - accuracy: 0.8781
 73/126 [================>.............] - ETA: 0s - loss: 0.3523 - accuracy: 0.8780
 85/126 [===================>..........] - ETA: 0s - loss: 0.3631 - accuracy: 0.8757
 97/126 [======================>.......] - ETA: 0s - loss: 0.3614 - accuracy: 0.8769
109/126 [========================>.....] - ETA: 0s - loss: 0.3655 - accuracy: 0.8761
121/126 [===========================>..] - ETA: 0s - loss: 0.3640 - accuracy: 0.8771
126/126 [==============================] - 1s 4ms/step - loss: 0.3635 - accuracy: 0.8774
Epoch 8/10

  1/126 [..............................] - ETA: 0s - loss: 0.3186 - accuracy: 0.8750
 13/126 [==>...........................] - ETA: 0s - loss: 0.3331 - accuracy: 0.8846
 25/126 [====>.........................] - ETA: 0s - loss: 0.3351 - accuracy: 0.8850
 37/126 [=======>......................] - ETA: 0s - loss: 0.3366 - accuracy: 0.8843
 49/126 [==========>...................] - ETA: 0s - loss: 0.3372 - accuracy: 0.8852
 61/126 [=============>................] - ETA: 0s - loss: 0.3334 - accuracy: 0.8842
 73/126 [================>.............] - ETA: 0s - loss: 0.3354 - accuracy: 0.8848
 85/126 [===================>..........] - ETA: 0s - loss: 0.3371 - accuracy: 0.8860
 97/126 [======================>.......] - ETA: 0s - loss: 0.3355 - accuracy: 0.8860
109/126 [========================>.....] - ETA: 0s - loss: 0.3349 - accuracy: 0.8845
121/126 [===========================>..] - ETA: 0s - loss: 0.3427 - accuracy: 0.8830
126/126 [==============================] - 1s 4ms/step - loss: 0.3457 - accuracy: 0.8826
Epoch 9/10

  1/126 [..............................] - ETA: 0s - loss: 0.2176 - accuracy: 0.9375
 13/126 [==>...........................] - ETA: 0s - loss: 0.3388 - accuracy: 0.8822
 25/126 [====>.........................] - ETA: 0s - loss: 0.3464 - accuracy: 0.8838
 37/126 [=======>......................] - ETA: 0s - loss: 0.3463 - accuracy: 0.8742
 49/126 [==========>...................] - ETA: 0s - loss: 0.3446 - accuracy: 0.8737
 61/126 [=============>................] - ETA: 0s - loss: 0.3297 - accuracy: 0.8811
 73/126 [================>.............] - ETA: 0s - loss: 0.3296 - accuracy: 0.8827
 85/126 [===================>..........] - ETA: 0s - loss: 0.3323 - accuracy: 0.8816
 97/126 [======================>.......] - ETA: 0s - loss: 0.3263 - accuracy: 0.8860
109/126 [========================>.....] - ETA: 0s - loss: 0.3284 - accuracy: 0.8873
121/126 [===========================>..] - ETA: 0s - loss: 0.3296 - accuracy: 0.8874
126/126 [==============================] - 1s 4ms/step - loss: 0.3285 - accuracy: 0.8878
Epoch 10/10

  1/126 [..............................] - ETA: 0s - loss: 0.3079 - accuracy: 0.8125
 13/126 [==>...........................] - ETA: 0s - loss: 0.2863 - accuracy: 0.9087
 25/126 [====>.........................] - ETA: 0s - loss: 0.2799 - accuracy: 0.9075
 37/126 [=======>......................] - ETA: 0s - loss: 0.2985 - accuracy: 0.9054
 49/126 [==========>...................] - ETA: 0s - loss: 0.2956 - accuracy: 0.9037
 67/126 [==============>...............] - ETA: 0s - loss: 0.2908 - accuracy: 0.9053
 79/126 [=================>............] - ETA: 0s - loss: 0.2832 - accuracy: 0.9066
 91/126 [====================>.........] - ETA: 0s - loss: 0.2922 - accuracy: 0.9052
103/126 [=======================>......] - ETA: 0s - loss: 0.2963 - accuracy: 0.9017
115/126 [==========================>...] - ETA: 0s - loss: 0.2983 - accuracy: 0.9000
126/126 [==============================] - 1s 4ms/step - loss: 0.3008 - accuracy: 0.8993


Datapoints:   0%|          | 0/33 [00:00<?, ?it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:   3%|▎         | 1/33 [00:00<00:08,  3.58it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:   6%|▌         | 2/33 [00:00<00:08,  3.46it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:   9%|▉         | 3/33 [00:00<00:08,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  12%|█▏        | 4/33 [00:01<00:08,  3.35it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  15%|█▌        | 5/33 [00:01<00:08,  3.36it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  18%|█▊        | 6/33 [00:01<00:08,  3.37it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  21%|██        | 7/33 [00:02<00:07,  3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  24%|██▍       | 8/33 [00:02<00:07,  3.46it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  27%|██▋       | 9/33 [00:02<00:06,  3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  30%|███       | 10/33 [00:02<00:06,  3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  33%|███▎      | 11/33 [00:03<00:06,  3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  36%|███▋      | 12/33 [00:03<00:06,  3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  39%|███▉      | 13/33 [00:03<00:05,  3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  42%|████▏     | 14/33 [00:04<00:05,  3.37it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  45%|████▌     | 15/33 [00:04<00:05,  3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  48%|████▊     | 16/33 [00:04<00:04,  3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  52%|█████▏    | 17/33 [00:04<00:04,  3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step


Datapoints:  55%|█████▍    | 18/33 [00:05<00:04,  3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  58%|█████▊    | 19/33 [00:05<00:04,  3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  61%|██████    | 20/33 [00:05<00:03,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  64%|██████▎   | 21/33 [00:06<00:03,  3.46it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  67%|██████▋   | 22/33 [00:06<00:03,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  70%|██████▉   | 23/33 [00:06<00:02,  3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  73%|███████▎  | 24/33 [00:07<00:02,  3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  76%|███████▌  | 25/33 [00:07<00:02,  3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  79%|███████▉  | 26/33 [00:07<00:02,  3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  82%|████████▏ | 27/33 [00:07<00:01,  3.46it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  85%|████████▍ | 28/33 [00:08<00:01,  3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  88%|████████▊ | 29/33 [00:08<00:01,  3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  91%|█████████ | 30/33 [00:08<00:00,  3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  94%|█████████▍| 31/33 [00:09<00:00,  3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints:  97%|█████████▋| 32/33 [00:09<00:00,  3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step


Datapoints: 100%|██████████| 33/33 [00:09<00:00,  3.39it/s]
Datapoints: 100%|██████████| 33/33 [00:09<00:00,  3.41it/s]

CV Folds: 100%|██████████| 3/3 [01:02<00:00, 20.17s/it]
CV Folds: 100%|██████████| 3/3 [01:02<00:00, 20.78s/it]

We can now look at the results per group:

cv_results["test_single_accuracy"]
[[0.85, 0.8666666666666667, 0.9333333333333333, 0.8, 0.85, 0.85, 0.85, 0.8833333333333333, 0.85, 0.85, 0.8166666666666667, 0.8666666666666667, 0.9333333333333333, 0.8, 0.8833333333333333, 0.8, 0.8, 0.8833333333333333, 0.7833333333333333, 0.8166666666666667, 0.8, 0.85, 0.8, 0.95, 0.8, 0.8833333333333333, 0.8833333333333333, 0.85, 0.85, 0.7833333333333333, 0.75, 0.8666666666666667, 0.8333333333333334, 0.9333333333333333], [0.7833333333333333, 0.8166666666666667, 0.7666666666666667, 0.7833333333333333, 0.85, 0.8, 0.8, 0.8166666666666667, 0.8, 0.8333333333333334, 0.8333333333333334, 0.8, 0.8166666666666667, 0.85, 0.75, 0.7, 0.8333333333333334, 0.8166666666666667, 0.8666666666666667, 0.7666666666666667, 0.7833333333333333, 0.7833333333333333, 0.7666666666666667, 0.8833333333333333, 0.8333333333333334, 0.7833333333333333, 0.8, 0.85, 0.85, 0.8166666666666667, 0.7666666666666667, 0.8, 0.8833333333333333], [0.7666666666666667, 0.9166666666666666, 0.8, 0.8, 0.7833333333333333, 0.85, 0.8833333333333333, 0.85, 0.9, 0.8833333333333333, 0.9, 0.85, 0.8833333333333333, 0.8166666666666667, 0.8333333333333334, 0.8333333333333334, 0.85, 0.7833333333333333, 0.7166666666666667, 0.7333333333333333, 0.75, 0.7833333333333333, 0.85, 0.7666666666666667, 0.7833333333333333, 0.9, 0.8666666666666667, 0.8333333333333334, 0.8333333333333334, 0.8166666666666667, 0.85, 0.85, 0.9]]

And the overall accuracy as the average over all samples of all groups within a fold:

cv_results["test_per_sample__accuracy"]
array([0.84705882, 0.80858586, 0.83080808])

Total running time of the script: (1 minutes 9.074 seconds)

Estimated memory usage: 278 MB

Gallery generated by Sphinx-Gallery