Note
Go to the end to download the full example code
Tensorflow/Keras#
Note
This example requires the tensorflow package to be installed.
Theoretically, tpcp is framework agnostic and can be used with any framework. However, due to the way some frameworks handle their objects, some special handling internally is required. Hence, this example does not only serve as example on how to use tensorflow with tpcp, but also as a test case for these special cases.
When using tpcp with any machine learning framework, you either want to use a pretrained model with a normal pipeline or a train your own model as part of an Optimizable Pipeline. Here we show the second case, as it is more complex, and you are likely able to figure out the first case yourself.
This means, we are planning to perform the following steps:
Create a pipeline that creates and trains a model.
Allow the modification of model hyperparameters.
Run a simple cross-validation to demonstrate the functionality.
This example reimplements the basic MNIST example from the [tensorflow documentation](https://www.tensorflow.org/tutorials/keras/classification).
Some Notes#
In this example we show how to implement a Pipeline that uses tensorflow. You could implement an Algorithm in a similar way. This would actually be easier, as no specific handling of the input data would be required. For a pipeline, we need to create a custom Dataset class, as this is the expected input for a pipeline.
The Dataset#
We are using the normal fashion MNIST dataset for this example It consists of 60.000 images of 28x28 pixels, each with a label. We will ignore the typical train-test split, as we want to do our own cross-validation.
In addition, we will simulate an additional “index level”. In this (and most typical deep learning datasets), each datapoint is one vector for which we can make one prediction. In tpcp, we usually deal with datasets, where you might have multiple pieces of information for each datapoint. For example, one datapoint could be a patient, for which we have an entire time series of measurements. We will simulate this here, by creating the index of our dataset as 1000 groups each containing 60 images.
Other than that, the dataset is pretty standard.
Besides the create_index method, we only need to implement the input_as_array and labels_as_array methods that
allow us to easily access the data once we selected a single group.
from functools import lru_cache
import numpy as np
import pandas as pd
import tensorflow as tf
from tpcp import Dataset
tf.keras.utils.set_random_seed(812)
tf.config.experimental.enable_op_determinism()
@lru_cache(maxsize=1)
def get_fashion_mnist_data():
# Note: We throw train and test sets together, as we don't care about the official split here.
# We will create our own split later.
(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.fashion_mnist.load_data()
return np.array(list(train_images) + list(test_images)), list(train_labels) + list(test_labels)
class FashionMNIST(Dataset):
def input_as_array(self) -> np.ndarray:
self.assert_is_single(None, "input_as_array")
group_id = int(self.group_label.group_id)
images, _ = get_fashion_mnist_data()
return images[group_id * 60 : (group_id + 1) * 60].reshape((60, 28, 28)) / 255
def labels_as_array(self) -> np.ndarray:
self.assert_is_single(None, "labels_as_array")
group_id = int(self.group_label.group_id)
_, labels = get_fashion_mnist_data()
return np.array(labels[group_id * 60 : (group_id + 1) * 60])
def create_index(self) -> pd.DataFrame:
# There are 60.000 images in total.
# We simulate 1000 groups of 60 images each.
return pd.DataFrame({"group_id": list(range(1000))})
We can see our Dataset works as expected:
dataset = FashionMNIST()
dataset[0].input_as_array().shape
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz
8192/29515 [=======>......................] - ETA: 0s
29515/29515 [==============================] - 0s 1us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz
8192/26421880 [..............................] - ETA: 0s
122880/26421880 [..............................] - ETA: 10s
909312/26421880 [>.............................] - ETA: 2s
5046272/26421880 [====>.........................] - ETA: 0s
9904128/26421880 [==========>...................] - ETA: 0s
14852096/26421880 [===============>..............] - ETA: 0s
19505152/26421880 [=====================>........] - ETA: 0s
24387584/26421880 [==========================>...] - ETA: 0s
26421880/26421880 [==============================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz
5148/5148 [==============================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz
8192/4422102 [..............................] - ETA: 0s
131072/4422102 [..............................] - ETA: 1s
671744/4422102 [===>..........................] - ETA: 0s
3997696/4422102 [==========================>...] - ETA: 0s
4422102/4422102 [==============================] - 0s 0us/step
(60, 28, 28)
dataset[0].labels_as_array().shape
(60,)
The Pipeline#
We will create a pipeline that uses a simple neural network to classify the images.
In tpcp, all “things” that should be optimized need to be parameters.
This means our model itself needs to be a parameter of the pipeline.
However, as we don’t have the model yet, as its creation depends on other hyperparameters, we add it as an optional
parameter initialized with None.
Further, we prefix the parameter name with an underscore, to signify, that this is not a parameter that should be
modified manually by the user.
This is just convention, and it is up to you to decide how you want to name your parameters.
We further introduce a hyperparameter n_dense_layer_nodes to show how we can influence the model creation.
The optimize method#
To make our pipeline optimizable, it needs to inherit from OptimizablePipeline.
Further we need to mark at least one of the parameters as OptiPara using the type annotation.
We do this for our _model parameter.
Finally, we need to implement the self_optimize method.
This method will get the entire training dataset as input and should update the _model parameter with the trained
model.
Hence, we first extract the relevant data (remember, each datapoint is 60 images), by concatinating all images over
all groups in the dataset.
Then we create the Keras model based on the hyperparameters.
Finally, we train the model and update the _model parameter.
Here we chose to wrap the method with make_optimize_safe.
This decorator will perform some runtime checks to ensure that the method is implemented correctly.
The run method#
The run method expects that the _model parameter is already set (i.e. the pipeline was already optimized).
It gets a single datapoint as input (remember, a datapoint is a single group of 60 images).
We then extract the data from the datapoint and let the model make a prediction.
We store the prediction on our output attribute predictions_.
The trailing underscore is a convention to signify, that this is an “result” attribute.
import warnings
from typing import Optional
from typing_extensions import Self
from tpcp import OptimizablePipeline, OptiPara, make_action_safe, make_optimize_safe
class KerasPipeline(OptimizablePipeline):
n_dense_layer_nodes: int
n_train_epochs: int
_model: OptiPara[Optional[tf.keras.Sequential]]
predictions_: np.ndarray
def __init__(self, n_dense_layer_nodes=128, n_train_epochs=5, _model: Optional[tf.keras.Sequential] = None):
self.n_dense_layer_nodes = n_dense_layer_nodes
self.n_train_epochs = n_train_epochs
self._model = _model
@property
def predicted_labels_(self):
return np.argmax(self.predictions_, axis=1)
@make_optimize_safe
def self_optimize(self, dataset, **_) -> Self:
data = np.vstack([d.input_as_array() for d in dataset])
labels = np.hstack([d.labels_as_array() for d in dataset])
print(data.shape)
if self._model is not None:
warnings.warn("Overwriting existing model!")
self._model = tf.keras.Sequential(
[
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(self.n_dense_layer_nodes, activation="relu"),
tf.keras.layers.Dense(10),
]
)
self._model.compile(
optimizer="adam",
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=["accuracy"],
)
self._model.fit(data, labels, epochs=self.n_train_epochs)
return self
@make_action_safe
def run(self, datapoint) -> Self:
if self._model is None:
raise RuntimeError("Model not trained yet!")
data = datapoint.input_as_array()
self.predictions_ = self._model.predict(data)
return self
Testing the pipeline#
We can now test our pipeline.
We will run the optimization using a couple of datapoints (to keep everything fast) and then use run to get the
predictions for a single unseen datapoint.
pipeline = KerasPipeline().self_optimize(FashionMNIST()[:10])
p1 = pipeline.run(FashionMNIST()[11])
print(p1.predicted_labels_)
print(FashionMNIST()[11].labels_as_array())
(600, 28, 28)
Epoch 1/5
1/19 [>.............................] - ETA: 19s - loss: 2.3593 - accuracy: 0.1250
13/19 [===================>..........] - ETA: 0s - loss: 1.6625 - accuracy: 0.4471
19/19 [==============================] - 1s 5ms/step - loss: 1.5112 - accuracy: 0.4933
Epoch 2/5
1/19 [>.............................] - ETA: 0s - loss: 0.9220 - accuracy: 0.7188
13/19 [===================>..........] - ETA: 0s - loss: 0.9038 - accuracy: 0.7115
19/19 [==============================] - 0s 5ms/step - loss: 0.8705 - accuracy: 0.7083
Epoch 3/5
1/19 [>.............................] - ETA: 0s - loss: 1.0179 - accuracy: 0.6875
13/19 [===================>..........] - ETA: 0s - loss: 0.6895 - accuracy: 0.7764
19/19 [==============================] - 0s 5ms/step - loss: 0.6799 - accuracy: 0.7850
Epoch 4/5
1/19 [>.............................] - ETA: 0s - loss: 0.4521 - accuracy: 0.8750
13/19 [===================>..........] - ETA: 0s - loss: 0.5492 - accuracy: 0.8197
19/19 [==============================] - 0s 5ms/step - loss: 0.5962 - accuracy: 0.8133
Epoch 5/5
1/19 [>.............................] - ETA: 0s - loss: 0.4230 - accuracy: 0.8438
13/19 [===================>..........] - ETA: 0s - loss: 0.4875 - accuracy: 0.8510
19/19 [==============================] - 0s 4ms/step - loss: 0.5158 - accuracy: 0.8400
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
[8 8 0 9 6 0 7 3 7 9 3 8 6 3 7 8 1 4 0 7 9 8 5 5 2 1 3 3 1 9 7 5 9 9 7 8 2
7 2 7 2 6 7 1 1 7 5 4 8 3 5 9 0 7 3 0 0 9 1 9]
[8 8 0 9 2 0 7 3 7 9 3 8 4 3 7 8 1 4 0 7 9 8 5 5 2 1 3 4 6 7 7 5 9 9 7 8 2
7 4 7 0 3 5 1 1 5 5 2 8 3 5 9 0 7 3 0 0 7 1 9]
We can see that even with just 5 epochs, the model already performs quite well. To quantify we can calculate the accuracy for this datapoint:
from sklearn.metrics import accuracy_score
accuracy_score(p1.predicted_labels_, FashionMNIST()[11].labels_as_array())
0.8
Cross Validation#
If we want to run a cross validation, we need to formalize the scoring into a function. We will calculate two types of accuracy: First, the accuracy per group and second, the accuracy over all images across all groups. For more information about how this works, check the Custom Scorer example.
from collections.abc import Sequence
from tpcp.validate import Aggregator
class SingleValueAccuracy(Aggregator[np.ndarray]):
RETURN_RAW_SCORES = False
@classmethod
def aggregate(cls, /, values: Sequence[tuple[np.ndarray, np.ndarray]], **_) -> dict[str, float]:
return {"accuracy": accuracy_score(np.hstack([v[0] for v in values]), np.hstack([v[1] for v in values]))}
def scoring(pipeline, datapoint):
result: np.ndarray = pipeline.safe_run(datapoint).predicted_labels_
reference = datapoint.labels_as_array()
return {
"accuracy": accuracy_score(result, reference),
"per_sample": SingleValueAccuracy((result, reference)),
}
Now we can run a cross validation. We will only run it on a subset of the data, to keep the runtime manageable.
Note
You might see warnings about retracing of the model. This is because we clone the pipeline before each call to the run method. This is a good idea to ensure that all pipelines are independent of each other, however, might result in some performance overhead.
from tpcp.optimize import Optimize
from tpcp.validate import cross_validate
pipeline = KerasPipeline(n_train_epochs=10)
cv_results = cross_validate(Optimize(pipeline), FashionMNIST()[:100], scoring=scoring, cv=3)
CV Folds: 0%| | 0/3 [00:00<?, ?it/s](3960, 28, 28)
Epoch 1/10
1/124 [..............................] - ETA: 1:55 - loss: 2.5074 - accuracy: 0.0938
12/124 [=>............................] - ETA: 0s - loss: 1.8208 - accuracy: 0.3125
24/124 [====>.........................] - ETA: 0s - loss: 1.4874 - accuracy: 0.4570
36/124 [=======>......................] - ETA: 0s - loss: 1.2993 - accuracy: 0.5278
48/124 [==========>...................] - ETA: 0s - loss: 1.1716 - accuracy: 0.5794
60/124 [=============>................] - ETA: 0s - loss: 1.0816 - accuracy: 0.6141
72/124 [================>.............] - ETA: 0s - loss: 1.0366 - accuracy: 0.6319
84/124 [===================>..........] - ETA: 0s - loss: 0.9932 - accuracy: 0.6484
96/124 [======================>.......] - ETA: 0s - loss: 0.9648 - accuracy: 0.6621
108/124 [=========================>....] - ETA: 0s - loss: 0.9299 - accuracy: 0.6756
120/124 [============================>.] - ETA: 0s - loss: 0.9033 - accuracy: 0.6883
124/124 [==============================] - 1s 5ms/step - loss: 0.8968 - accuracy: 0.6919
Epoch 2/10
1/124 [..............................] - ETA: 0s - loss: 1.0481 - accuracy: 0.7500
13/124 [==>...........................] - ETA: 0s - loss: 0.6082 - accuracy: 0.8005
24/124 [====>.........................] - ETA: 0s - loss: 0.5817 - accuracy: 0.8112
36/124 [=======>......................] - ETA: 0s - loss: 0.6109 - accuracy: 0.7925
48/124 [==========>...................] - ETA: 0s - loss: 0.5921 - accuracy: 0.8008
61/124 [=============>................] - ETA: 0s - loss: 0.5840 - accuracy: 0.7976
73/124 [================>.............] - ETA: 0s - loss: 0.5668 - accuracy: 0.8074
85/124 [===================>..........] - ETA: 0s - loss: 0.5685 - accuracy: 0.8062
97/124 [======================>.......] - ETA: 0s - loss: 0.5629 - accuracy: 0.8115
109/124 [=========================>....] - ETA: 0s - loss: 0.5591 - accuracy: 0.8114
121/124 [============================>.] - ETA: 0s - loss: 0.5576 - accuracy: 0.8122
124/124 [==============================] - 1s 4ms/step - loss: 0.5606 - accuracy: 0.8104
Epoch 3/10
1/124 [..............................] - ETA: 0s - loss: 0.7180 - accuracy: 0.6250
13/124 [==>...........................] - ETA: 0s - loss: 0.5132 - accuracy: 0.8053
25/124 [=====>........................] - ETA: 0s - loss: 0.5200 - accuracy: 0.8125
37/124 [=======>......................] - ETA: 0s - loss: 0.5243 - accuracy: 0.8218
49/124 [==========>...................] - ETA: 0s - loss: 0.5135 - accuracy: 0.8233
61/124 [=============>................] - ETA: 0s - loss: 0.5053 - accuracy: 0.8222
73/124 [================>.............] - ETA: 0s - loss: 0.5060 - accuracy: 0.8215
85/124 [===================>..........] - ETA: 0s - loss: 0.4954 - accuracy: 0.8257
97/124 [======================>.......] - ETA: 0s - loss: 0.4940 - accuracy: 0.8264
109/124 [=========================>....] - ETA: 0s - loss: 0.4974 - accuracy: 0.8243
122/124 [============================>.] - ETA: 0s - loss: 0.4976 - accuracy: 0.8240
124/124 [==============================] - 1s 4ms/step - loss: 0.5009 - accuracy: 0.8232
Epoch 4/10
1/124 [..............................] - ETA: 0s - loss: 0.4630 - accuracy: 0.8125
13/124 [==>...........................] - ETA: 0s - loss: 0.3962 - accuracy: 0.8630
25/124 [=====>........................] - ETA: 0s - loss: 0.4104 - accuracy: 0.8587
37/124 [=======>......................] - ETA: 0s - loss: 0.4127 - accuracy: 0.8556
48/124 [==========>...................] - ETA: 0s - loss: 0.4300 - accuracy: 0.8522
58/124 [=============>................] - ETA: 0s - loss: 0.4337 - accuracy: 0.8491
69/124 [===============>..............] - ETA: 0s - loss: 0.4418 - accuracy: 0.8478
81/124 [==================>...........] - ETA: 0s - loss: 0.4444 - accuracy: 0.8484
93/124 [=====================>........] - ETA: 0s - loss: 0.4434 - accuracy: 0.8464
105/124 [========================>.....] - ETA: 0s - loss: 0.4405 - accuracy: 0.8467
117/124 [===========================>..] - ETA: 0s - loss: 0.4450 - accuracy: 0.8438
124/124 [==============================] - 1s 5ms/step - loss: 0.4448 - accuracy: 0.8437
Epoch 5/10
1/124 [..............................] - ETA: 0s - loss: 0.6924 - accuracy: 0.6250
13/124 [==>...........................] - ETA: 0s - loss: 0.4504 - accuracy: 0.8173
25/124 [=====>........................] - ETA: 0s - loss: 0.4423 - accuracy: 0.8263
37/124 [=======>......................] - ETA: 0s - loss: 0.4455 - accuracy: 0.8294
49/124 [==========>...................] - ETA: 0s - loss: 0.4360 - accuracy: 0.8399
61/124 [=============>................] - ETA: 0s - loss: 0.4225 - accuracy: 0.8438
73/124 [================>.............] - ETA: 0s - loss: 0.4221 - accuracy: 0.8485
85/124 [===================>..........] - ETA: 0s - loss: 0.4100 - accuracy: 0.8540
97/124 [======================>.......] - ETA: 0s - loss: 0.4067 - accuracy: 0.8560
109/124 [=========================>....] - ETA: 0s - loss: 0.4122 - accuracy: 0.8544
121/124 [============================>.] - ETA: 0s - loss: 0.4119 - accuracy: 0.8546
124/124 [==============================] - 1s 4ms/step - loss: 0.4119 - accuracy: 0.8540
Epoch 6/10
1/124 [..............................] - ETA: 0s - loss: 0.5976 - accuracy: 0.7812
14/124 [==>...........................] - ETA: 0s - loss: 0.3864 - accuracy: 0.8705
26/124 [=====>........................] - ETA: 0s - loss: 0.4092 - accuracy: 0.8558
38/124 [========>.....................] - ETA: 0s - loss: 0.3782 - accuracy: 0.8660
50/124 [===========>..................] - ETA: 0s - loss: 0.3883 - accuracy: 0.8650
62/124 [==============>...............] - ETA: 0s - loss: 0.3825 - accuracy: 0.8669
74/124 [================>.............] - ETA: 0s - loss: 0.3824 - accuracy: 0.8657
86/124 [===================>..........] - ETA: 0s - loss: 0.3877 - accuracy: 0.8645
98/124 [======================>.......] - ETA: 0s - loss: 0.3790 - accuracy: 0.8686
110/124 [=========================>....] - ETA: 0s - loss: 0.3823 - accuracy: 0.8673
122/124 [============================>.] - ETA: 0s - loss: 0.3810 - accuracy: 0.8678
124/124 [==============================] - 1s 4ms/step - loss: 0.3792 - accuracy: 0.8687
Epoch 7/10
1/124 [..............................] - ETA: 0s - loss: 0.4009 - accuracy: 0.8438
13/124 [==>...........................] - ETA: 0s - loss: 0.3694 - accuracy: 0.8726
25/124 [=====>........................] - ETA: 0s - loss: 0.3564 - accuracy: 0.8700
37/124 [=======>......................] - ETA: 0s - loss: 0.3387 - accuracy: 0.8758
49/124 [==========>...................] - ETA: 0s - loss: 0.3485 - accuracy: 0.8744
61/124 [=============>................] - ETA: 0s - loss: 0.3571 - accuracy: 0.8760
73/124 [================>.............] - ETA: 0s - loss: 0.3593 - accuracy: 0.8750
85/124 [===================>..........] - ETA: 0s - loss: 0.3578 - accuracy: 0.8757
97/124 [======================>.......] - ETA: 0s - loss: 0.3618 - accuracy: 0.8766
109/124 [=========================>....] - ETA: 0s - loss: 0.3580 - accuracy: 0.8767
121/124 [============================>.] - ETA: 0s - loss: 0.3582 - accuracy: 0.8755
124/124 [==============================] - 1s 4ms/step - loss: 0.3555 - accuracy: 0.8768
Epoch 8/10
1/124 [..............................] - ETA: 0s - loss: 0.2810 - accuracy: 0.9062
13/124 [==>...........................] - ETA: 0s - loss: 0.3236 - accuracy: 0.8966
25/124 [=====>........................] - ETA: 0s - loss: 0.2929 - accuracy: 0.9150
37/124 [=======>......................] - ETA: 0s - loss: 0.2969 - accuracy: 0.9079
49/124 [==========>...................] - ETA: 0s - loss: 0.3031 - accuracy: 0.9024
61/124 [=============>................] - ETA: 0s - loss: 0.3088 - accuracy: 0.8981
73/124 [================>.............] - ETA: 0s - loss: 0.3173 - accuracy: 0.8943
85/124 [===================>..........] - ETA: 0s - loss: 0.3251 - accuracy: 0.8926
97/124 [======================>.......] - ETA: 0s - loss: 0.3274 - accuracy: 0.8914
109/124 [=========================>....] - ETA: 0s - loss: 0.3369 - accuracy: 0.8879
121/124 [============================>.] - ETA: 0s - loss: 0.3466 - accuracy: 0.8825
124/124 [==============================] - 1s 4ms/step - loss: 0.3464 - accuracy: 0.8826
Epoch 9/10
1/124 [..............................] - ETA: 0s - loss: 0.2351 - accuracy: 0.9375
13/124 [==>...........................] - ETA: 0s - loss: 0.3713 - accuracy: 0.8726
25/124 [=====>........................] - ETA: 0s - loss: 0.3446 - accuracy: 0.8850
37/124 [=======>......................] - ETA: 0s - loss: 0.3441 - accuracy: 0.8742
49/124 [==========>...................] - ETA: 0s - loss: 0.3385 - accuracy: 0.8801
60/124 [=============>................] - ETA: 0s - loss: 0.3325 - accuracy: 0.8828
70/124 [===============>..............] - ETA: 0s - loss: 0.3258 - accuracy: 0.8862
82/124 [==================>...........] - ETA: 0s - loss: 0.3134 - accuracy: 0.8921
94/124 [=====================>........] - ETA: 0s - loss: 0.3241 - accuracy: 0.8893
106/124 [========================>.....] - ETA: 0s - loss: 0.3215 - accuracy: 0.8912
118/124 [===========================>..] - ETA: 0s - loss: 0.3186 - accuracy: 0.8893
124/124 [==============================] - 1s 5ms/step - loss: 0.3196 - accuracy: 0.8886
Epoch 10/10
1/124 [..............................] - ETA: 0s - loss: 0.2527 - accuracy: 0.9688
13/124 [==>...........................] - ETA: 0s - loss: 0.2215 - accuracy: 0.9255
25/124 [=====>........................] - ETA: 0s - loss: 0.2457 - accuracy: 0.9125
37/124 [=======>......................] - ETA: 0s - loss: 0.2692 - accuracy: 0.9037
49/124 [==========>...................] - ETA: 0s - loss: 0.2755 - accuracy: 0.9031
61/124 [=============>................] - ETA: 0s - loss: 0.2910 - accuracy: 0.9001
73/124 [================>.............] - ETA: 0s - loss: 0.2991 - accuracy: 0.8951
85/124 [===================>..........] - ETA: 0s - loss: 0.2970 - accuracy: 0.8960
97/124 [======================>.......] - ETA: 0s - loss: 0.2953 - accuracy: 0.8982
109/124 [=========================>....] - ETA: 0s - loss: 0.2941 - accuracy: 0.8999
121/124 [============================>.] - ETA: 0s - loss: 0.2929 - accuracy: 0.8993
124/124 [==============================] - 1s 4ms/step - loss: 0.2921 - accuracy: 0.8997
Datapoints: 0%| | 0/34 [00:00<?, ?it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 3%|▎ | 1/34 [00:00<00:09, 3.54it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 6%|▌ | 2/34 [00:00<00:09, 3.46it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 9%|▉ | 3/34 [00:00<00:08, 3.47it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 12%|█▏ | 4/34 [00:01<00:08, 3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 15%|█▍ | 5/34 [00:01<00:08, 3.48it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 18%|█▊ | 6/34 [00:01<00:08, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 21%|██ | 7/34 [00:02<00:07, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 24%|██▎ | 8/34 [00:02<00:07, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 26%|██▋ | 9/34 [00:02<00:07, 3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 29%|██▉ | 10/34 [00:02<00:07, 3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 32%|███▏ | 11/34 [00:03<00:06, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 35%|███▌ | 12/34 [00:03<00:06, 3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 38%|███▊ | 13/34 [00:03<00:06, 3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 41%|████ | 14/34 [00:04<00:05, 3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 44%|████▍ | 15/34 [00:04<00:05, 3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 47%|████▋ | 16/34 [00:04<00:05, 3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 50%|█████ | 17/34 [00:04<00:04, 3.47it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 53%|█████▎ | 18/34 [00:05<00:04, 3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 56%|█████▌ | 19/34 [00:05<00:04, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 59%|█████▉ | 20/34 [00:05<00:04, 3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 62%|██████▏ | 21/34 [00:06<00:03, 3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 65%|██████▍ | 22/34 [00:06<00:03, 3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 68%|██████▊ | 23/34 [00:06<00:03, 3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 71%|███████ | 24/34 [00:07<00:02, 3.37it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 74%|███████▎ | 25/34 [00:07<00:02, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 76%|███████▋ | 26/34 [00:07<00:02, 3.47it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 79%|███████▉ | 27/34 [00:07<00:02, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 82%|████████▏ | 28/34 [00:08<00:01, 3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 85%|████████▌ | 29/34 [00:08<00:01, 3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 88%|████████▊ | 30/34 [00:08<00:01, 3.37it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 91%|█████████ | 31/34 [00:09<00:00, 3.37it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 94%|█████████▍| 32/34 [00:09<00:00, 3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 97%|█████████▋| 33/34 [00:09<00:00, 3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 100%|██████████| 34/34 [00:09<00:00, 3.39it/s]
Datapoints: 100%|██████████| 34/34 [00:09<00:00, 3.41it/s]
CV Folds: 33%|███▎ | 1/3 [00:22<00:45, 22.57s/it](4020, 28, 28)
Epoch 1/10
1/126 [..............................] - ETA: 1:55 - loss: 2.3656 - accuracy: 0.1250
13/126 [==>...........................] - ETA: 0s - loss: 1.7826 - accuracy: 0.3870
25/126 [====>.........................] - ETA: 0s - loss: 1.4578 - accuracy: 0.5150
37/126 [=======>......................] - ETA: 0s - loss: 1.3063 - accuracy: 0.5633
49/126 [==========>...................] - ETA: 0s - loss: 1.2005 - accuracy: 0.6014
61/126 [=============>................] - ETA: 0s - loss: 1.1294 - accuracy: 0.6168
73/126 [================>.............] - ETA: 0s - loss: 1.0706 - accuracy: 0.6378
85/126 [===================>..........] - ETA: 0s - loss: 1.0088 - accuracy: 0.6603
97/126 [======================>.......] - ETA: 0s - loss: 0.9718 - accuracy: 0.6727
109/126 [========================>.....] - ETA: 0s - loss: 0.9331 - accuracy: 0.6841
121/126 [===========================>..] - ETA: 0s - loss: 0.9028 - accuracy: 0.6955
126/126 [==============================] - 1s 4ms/step - loss: 0.8913 - accuracy: 0.7000
Epoch 2/10
1/126 [..............................] - ETA: 0s - loss: 0.4984 - accuracy: 0.8125
13/126 [==>...........................] - ETA: 0s - loss: 0.5529 - accuracy: 0.7957
25/126 [====>.........................] - ETA: 0s - loss: 0.5682 - accuracy: 0.8037
37/126 [=======>......................] - ETA: 0s - loss: 0.5658 - accuracy: 0.7990
49/126 [==========>...................] - ETA: 0s - loss: 0.5778 - accuracy: 0.7927
61/126 [=============>................] - ETA: 0s - loss: 0.5962 - accuracy: 0.7874
73/126 [================>.............] - ETA: 0s - loss: 0.5902 - accuracy: 0.7902
85/126 [===================>..........] - ETA: 0s - loss: 0.5773 - accuracy: 0.7926
97/126 [======================>.......] - ETA: 0s - loss: 0.5772 - accuracy: 0.7938
109/126 [========================>.....] - ETA: 0s - loss: 0.5713 - accuracy: 0.7967
121/126 [===========================>..] - ETA: 0s - loss: 0.5649 - accuracy: 0.7998
126/126 [==============================] - 1s 4ms/step - loss: 0.5662 - accuracy: 0.8002
Epoch 3/10
1/126 [..............................] - ETA: 0s - loss: 0.5249 - accuracy: 0.8125
13/126 [==>...........................] - ETA: 0s - loss: 0.5380 - accuracy: 0.8125
25/126 [====>.........................] - ETA: 0s - loss: 0.5201 - accuracy: 0.8275
37/126 [=======>......................] - ETA: 0s - loss: 0.4840 - accuracy: 0.8353
49/126 [==========>...................] - ETA: 0s - loss: 0.4717 - accuracy: 0.8329
61/126 [=============>................] - ETA: 0s - loss: 0.4678 - accuracy: 0.8381
73/126 [================>.............] - ETA: 0s - loss: 0.4692 - accuracy: 0.8373
85/126 [===================>..........] - ETA: 0s - loss: 0.4810 - accuracy: 0.8342
97/126 [======================>.......] - ETA: 0s - loss: 0.4782 - accuracy: 0.8354
109/126 [========================>.....] - ETA: 0s - loss: 0.4805 - accuracy: 0.8343
121/126 [===========================>..] - ETA: 0s - loss: 0.4901 - accuracy: 0.8275
126/126 [==============================] - 1s 4ms/step - loss: 0.4893 - accuracy: 0.8279
Epoch 4/10
1/126 [..............................] - ETA: 0s - loss: 0.2082 - accuracy: 0.9688
13/126 [==>...........................] - ETA: 0s - loss: 0.4042 - accuracy: 0.8654
25/126 [====>.........................] - ETA: 0s - loss: 0.4238 - accuracy: 0.8587
37/126 [=======>......................] - ETA: 0s - loss: 0.4437 - accuracy: 0.8480
49/126 [==========>...................] - ETA: 0s - loss: 0.4393 - accuracy: 0.8469
61/126 [=============>................] - ETA: 0s - loss: 0.4340 - accuracy: 0.8478
73/126 [================>.............] - ETA: 0s - loss: 0.4283 - accuracy: 0.8527
85/126 [===================>..........] - ETA: 0s - loss: 0.4348 - accuracy: 0.8526
97/126 [======================>.......] - ETA: 0s - loss: 0.4394 - accuracy: 0.8502
109/126 [========================>.....] - ETA: 0s - loss: 0.4472 - accuracy: 0.8438
121/126 [===========================>..] - ETA: 0s - loss: 0.4457 - accuracy: 0.8450
126/126 [==============================] - 1s 4ms/step - loss: 0.4431 - accuracy: 0.8453
Epoch 5/10
1/126 [..............................] - ETA: 0s - loss: 0.1979 - accuracy: 0.9375
13/126 [==>...........................] - ETA: 0s - loss: 0.3542 - accuracy: 0.8822
25/126 [====>.........................] - ETA: 0s - loss: 0.3946 - accuracy: 0.8612
37/126 [=======>......................] - ETA: 0s - loss: 0.4174 - accuracy: 0.8471
49/126 [==========>...................] - ETA: 0s - loss: 0.4057 - accuracy: 0.8540
61/126 [=============>................] - ETA: 0s - loss: 0.4065 - accuracy: 0.8535
73/126 [================>.............] - ETA: 0s - loss: 0.4017 - accuracy: 0.8570
85/126 [===================>..........] - ETA: 0s - loss: 0.4044 - accuracy: 0.8570
97/126 [======================>.......] - ETA: 0s - loss: 0.4087 - accuracy: 0.8544
109/126 [========================>.....] - ETA: 0s - loss: 0.4048 - accuracy: 0.8569
121/126 [===========================>..] - ETA: 0s - loss: 0.4056 - accuracy: 0.8574
126/126 [==============================] - 1s 4ms/step - loss: 0.4046 - accuracy: 0.8580
Epoch 6/10
1/126 [..............................] - ETA: 0s - loss: 0.4598 - accuracy: 0.8125
13/126 [==>...........................] - ETA: 0s - loss: 0.3434 - accuracy: 0.8534
25/126 [====>.........................] - ETA: 0s - loss: 0.3639 - accuracy: 0.8625
37/126 [=======>......................] - ETA: 0s - loss: 0.3791 - accuracy: 0.8573
49/126 [==========>...................] - ETA: 0s - loss: 0.3696 - accuracy: 0.8642
61/126 [=============>................] - ETA: 0s - loss: 0.3709 - accuracy: 0.8622
73/126 [================>.............] - ETA: 0s - loss: 0.3741 - accuracy: 0.8647
85/126 [===================>..........] - ETA: 0s - loss: 0.3770 - accuracy: 0.8654
97/126 [======================>.......] - ETA: 0s - loss: 0.3796 - accuracy: 0.8650
109/126 [========================>.....] - ETA: 0s - loss: 0.3698 - accuracy: 0.8687
121/126 [===========================>..] - ETA: 0s - loss: 0.3775 - accuracy: 0.8662
126/126 [==============================] - 1s 4ms/step - loss: 0.3772 - accuracy: 0.8667
Epoch 7/10
1/126 [..............................] - ETA: 0s - loss: 0.2244 - accuracy: 0.9062
13/126 [==>...........................] - ETA: 0s - loss: 0.3541 - accuracy: 0.8798
25/126 [====>.........................] - ETA: 0s - loss: 0.3532 - accuracy: 0.8813
37/126 [=======>......................] - ETA: 0s - loss: 0.3349 - accuracy: 0.8843
49/126 [==========>...................] - ETA: 0s - loss: 0.3264 - accuracy: 0.8884
61/126 [=============>................] - ETA: 0s - loss: 0.3308 - accuracy: 0.8858
73/126 [================>.............] - ETA: 0s - loss: 0.3448 - accuracy: 0.8823
85/126 [===================>..........] - ETA: 0s - loss: 0.3522 - accuracy: 0.8790
97/126 [======================>.......] - ETA: 0s - loss: 0.3469 - accuracy: 0.8798
109/126 [========================>.....] - ETA: 0s - loss: 0.3480 - accuracy: 0.8802
121/126 [===========================>..] - ETA: 0s - loss: 0.3487 - accuracy: 0.8786
126/126 [==============================] - 1s 4ms/step - loss: 0.3479 - accuracy: 0.8786
Epoch 8/10
1/126 [..............................] - ETA: 0s - loss: 0.2092 - accuracy: 0.9375
13/126 [==>...........................] - ETA: 0s - loss: 0.3092 - accuracy: 0.8942
25/126 [====>.........................] - ETA: 0s - loss: 0.2997 - accuracy: 0.9000
37/126 [=======>......................] - ETA: 0s - loss: 0.2936 - accuracy: 0.8986
49/126 [==========>...................] - ETA: 0s - loss: 0.3115 - accuracy: 0.8935
61/126 [=============>................] - ETA: 0s - loss: 0.3090 - accuracy: 0.8934
73/126 [================>.............] - ETA: 0s - loss: 0.3184 - accuracy: 0.8887
85/126 [===================>..........] - ETA: 0s - loss: 0.3252 - accuracy: 0.8875
97/126 [======================>.......] - ETA: 0s - loss: 0.3264 - accuracy: 0.8869
109/126 [========================>.....] - ETA: 0s - loss: 0.3268 - accuracy: 0.8885
121/126 [===========================>..] - ETA: 0s - loss: 0.3287 - accuracy: 0.8861
126/126 [==============================] - 1s 4ms/step - loss: 0.3285 - accuracy: 0.8863
Epoch 9/10
1/126 [..............................] - ETA: 0s - loss: 0.1886 - accuracy: 0.9062
13/126 [==>...........................] - ETA: 0s - loss: 0.2965 - accuracy: 0.9038
25/126 [====>.........................] - ETA: 0s - loss: 0.2998 - accuracy: 0.9000
37/126 [=======>......................] - ETA: 0s - loss: 0.2976 - accuracy: 0.8953
49/126 [==========>...................] - ETA: 0s - loss: 0.3057 - accuracy: 0.8909
61/126 [=============>................] - ETA: 0s - loss: 0.3055 - accuracy: 0.8842
73/126 [================>.............] - ETA: 0s - loss: 0.3189 - accuracy: 0.8789
85/126 [===================>..........] - ETA: 0s - loss: 0.3181 - accuracy: 0.8809
97/126 [======================>.......] - ETA: 0s - loss: 0.3062 - accuracy: 0.8860
109/126 [========================>.....] - ETA: 0s - loss: 0.3139 - accuracy: 0.8862
121/126 [===========================>..] - ETA: 0s - loss: 0.3122 - accuracy: 0.8892
126/126 [==============================] - 1s 4ms/step - loss: 0.3120 - accuracy: 0.8900
Epoch 10/10
1/126 [..............................] - ETA: 0s - loss: 0.2327 - accuracy: 0.9062
13/126 [==>...........................] - ETA: 0s - loss: 0.2908 - accuracy: 0.8870
25/126 [====>.........................] - ETA: 0s - loss: 0.2960 - accuracy: 0.8825
37/126 [=======>......................] - ETA: 0s - loss: 0.3081 - accuracy: 0.8851
49/126 [==========>...................] - ETA: 0s - loss: 0.2985 - accuracy: 0.8903
61/126 [=============>................] - ETA: 0s - loss: 0.2978 - accuracy: 0.8888
73/126 [================>.............] - ETA: 0s - loss: 0.3040 - accuracy: 0.8883
85/126 [===================>..........] - ETA: 0s - loss: 0.3004 - accuracy: 0.8886
97/126 [======================>.......] - ETA: 0s - loss: 0.3038 - accuracy: 0.8876
109/126 [========================>.....] - ETA: 0s - loss: 0.3052 - accuracy: 0.8882
121/126 [===========================>..] - ETA: 0s - loss: 0.3039 - accuracy: 0.8882
126/126 [==============================] - 1s 4ms/step - loss: 0.3065 - accuracy: 0.8871
Datapoints: 0%| | 0/33 [00:00<?, ?it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 3%|▎ | 1/33 [00:00<00:08, 3.58it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step
Datapoints: 6%|▌ | 2/33 [00:00<00:08, 3.48it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 9%|▉ | 3/33 [00:00<00:08, 3.45it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 12%|█▏ | 4/33 [00:01<00:08, 3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 15%|█▌ | 5/33 [00:01<00:08, 3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 18%|█▊ | 6/33 [00:01<00:07, 3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 21%|██ | 7/33 [00:02<00:07, 3.45it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 24%|██▍ | 8/33 [00:02<00:07, 3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 27%|██▋ | 9/33 [00:02<00:07, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 30%|███ | 10/33 [00:02<00:06, 3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 33%|███▎ | 11/33 [00:03<00:06, 3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 36%|███▋ | 12/33 [00:03<00:06, 3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 39%|███▉ | 13/33 [00:03<00:05, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step
Datapoints: 42%|████▏ | 14/33 [00:04<00:05, 3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 45%|████▌ | 15/33 [00:04<00:05, 3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 48%|████▊ | 16/33 [00:04<00:05, 3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 52%|█████▏ | 17/33 [00:04<00:04, 3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 55%|█████▍ | 18/33 [00:05<00:04, 3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 58%|█████▊ | 19/33 [00:05<00:04, 3.45it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step
Datapoints: 61%|██████ | 20/33 [00:05<00:03, 3.48it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 64%|██████▎ | 21/33 [00:06<00:03, 3.52it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 67%|██████▋ | 22/33 [00:06<00:03, 3.45it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 70%|██████▉ | 23/33 [00:06<00:02, 3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 73%|███████▎ | 24/33 [00:06<00:02, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step
Datapoints: 76%|███████▌ | 25/33 [00:07<00:02, 3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 79%|███████▉ | 26/33 [00:07<00:02, 3.48it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 82%|████████▏ | 27/33 [00:07<00:01, 3.51it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 85%|████████▍ | 28/33 [00:08<00:01, 3.45it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 88%|████████▊ | 29/33 [00:08<00:01, 3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 91%|█████████ | 30/33 [00:08<00:00, 3.48it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 94%|█████████▍| 31/33 [00:09<00:00, 3.48it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 97%|█████████▋| 32/33 [00:09<00:00, 3.51it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 100%|██████████| 33/33 [00:09<00:00, 3.52it/s]
Datapoints: 100%|██████████| 33/33 [00:09<00:00, 3.45it/s]
CV Folds: 67%|██████▋ | 2/3 [00:44<00:22, 22.31s/it](4020, 28, 28)
Epoch 1/10
1/126 [..............................] - ETA: 1:56 - loss: 2.3129 - accuracy: 0.0938
12/126 [=>............................] - ETA: 0s - loss: 1.7505 - accuracy: 0.3984
24/126 [====>.........................] - ETA: 0s - loss: 1.4718 - accuracy: 0.5026
36/126 [=======>......................] - ETA: 0s - loss: 1.3248 - accuracy: 0.5460
48/126 [==========>...................] - ETA: 0s - loss: 1.2118 - accuracy: 0.5905
60/126 [=============>................] - ETA: 0s - loss: 1.1265 - accuracy: 0.6182
72/126 [================>.............] - ETA: 0s - loss: 1.0826 - accuracy: 0.6328
84/126 [===================>..........] - ETA: 0s - loss: 1.0169 - accuracy: 0.6562
96/126 [=====================>........] - ETA: 0s - loss: 0.9770 - accuracy: 0.6712
108/126 [========================>.....] - ETA: 0s - loss: 0.9428 - accuracy: 0.6808
120/126 [===========================>..] - ETA: 0s - loss: 0.9144 - accuracy: 0.6922
126/126 [==============================] - 1s 4ms/step - loss: 0.8965 - accuracy: 0.6983
Epoch 2/10
1/126 [..............................] - ETA: 0s - loss: 0.4769 - accuracy: 0.8125
13/126 [==>...........................] - ETA: 0s - loss: 0.5761 - accuracy: 0.7957
25/126 [====>.........................] - ETA: 0s - loss: 0.6120 - accuracy: 0.7912
37/126 [=======>......................] - ETA: 0s - loss: 0.6063 - accuracy: 0.7889
49/126 [==========>...................] - ETA: 0s - loss: 0.5931 - accuracy: 0.7972
61/126 [=============>................] - ETA: 0s - loss: 0.5958 - accuracy: 0.7935
73/126 [================>.............] - ETA: 0s - loss: 0.5951 - accuracy: 0.7954
85/126 [===================>..........] - ETA: 0s - loss: 0.5894 - accuracy: 0.7967
97/126 [======================>.......] - ETA: 0s - loss: 0.5817 - accuracy: 0.8015
109/126 [========================>.....] - ETA: 0s - loss: 0.5710 - accuracy: 0.8048
121/126 [===========================>..] - ETA: 0s - loss: 0.5654 - accuracy: 0.8068
126/126 [==============================] - 1s 4ms/step - loss: 0.5750 - accuracy: 0.8040
Epoch 3/10
1/126 [..............................] - ETA: 0s - loss: 0.6893 - accuracy: 0.7812
13/126 [==>...........................] - ETA: 0s - loss: 0.5276 - accuracy: 0.8245
25/126 [====>.........................] - ETA: 0s - loss: 0.5681 - accuracy: 0.8075
37/126 [=======>......................] - ETA: 0s - loss: 0.5517 - accuracy: 0.8083
49/126 [==========>...................] - ETA: 0s - loss: 0.5377 - accuracy: 0.8119
61/126 [=============>................] - ETA: 0s - loss: 0.5310 - accuracy: 0.8166
73/126 [================>.............] - ETA: 0s - loss: 0.5269 - accuracy: 0.8185
85/126 [===================>..........] - ETA: 0s - loss: 0.5284 - accuracy: 0.8210
97/126 [======================>.......] - ETA: 0s - loss: 0.5211 - accuracy: 0.8212
109/126 [========================>.....] - ETA: 0s - loss: 0.5114 - accuracy: 0.8248
121/126 [===========================>..] - ETA: 0s - loss: 0.5134 - accuracy: 0.8236
126/126 [==============================] - 1s 4ms/step - loss: 0.5140 - accuracy: 0.8224
Epoch 4/10
1/126 [..............................] - ETA: 0s - loss: 0.4519 - accuracy: 0.9062
13/126 [==>...........................] - ETA: 0s - loss: 0.4019 - accuracy: 0.8750
25/126 [====>.........................] - ETA: 0s - loss: 0.4193 - accuracy: 0.8587
37/126 [=======>......................] - ETA: 0s - loss: 0.4405 - accuracy: 0.8480
49/126 [==========>...................] - ETA: 0s - loss: 0.4532 - accuracy: 0.8399
61/126 [=============>................] - ETA: 0s - loss: 0.4511 - accuracy: 0.8412
73/126 [================>.............] - ETA: 0s - loss: 0.4450 - accuracy: 0.8438
85/126 [===================>..........] - ETA: 0s - loss: 0.4464 - accuracy: 0.8438
97/126 [======================>.......] - ETA: 0s - loss: 0.4548 - accuracy: 0.8415
109/126 [========================>.....] - ETA: 0s - loss: 0.4619 - accuracy: 0.8394
121/126 [===========================>..] - ETA: 0s - loss: 0.4629 - accuracy: 0.8383
126/126 [==============================] - 1s 4ms/step - loss: 0.4667 - accuracy: 0.8358
Epoch 5/10
1/126 [..............................] - ETA: 0s - loss: 0.4192 - accuracy: 0.8125
13/126 [==>...........................] - ETA: 0s - loss: 0.4263 - accuracy: 0.8654
25/126 [====>.........................] - ETA: 0s - loss: 0.4182 - accuracy: 0.8625
37/126 [=======>......................] - ETA: 0s - loss: 0.4131 - accuracy: 0.8556
49/126 [==========>...................] - ETA: 0s - loss: 0.4216 - accuracy: 0.8635
61/126 [=============>................] - ETA: 0s - loss: 0.4160 - accuracy: 0.8648
73/126 [================>.............] - ETA: 0s - loss: 0.4160 - accuracy: 0.8634
85/126 [===================>..........] - ETA: 0s - loss: 0.4222 - accuracy: 0.8599
97/126 [======================>.......] - ETA: 0s - loss: 0.4279 - accuracy: 0.8595
109/126 [========================>.....] - ETA: 0s - loss: 0.4202 - accuracy: 0.8595
121/126 [===========================>..] - ETA: 0s - loss: 0.4177 - accuracy: 0.8598
126/126 [==============================] - 1s 4ms/step - loss: 0.4173 - accuracy: 0.8600
Epoch 6/10
1/126 [..............................] - ETA: 0s - loss: 0.8799 - accuracy: 0.7500
13/126 [==>...........................] - ETA: 0s - loss: 0.3963 - accuracy: 0.8582
25/126 [====>.........................] - ETA: 0s - loss: 0.3716 - accuracy: 0.8662
37/126 [=======>......................] - ETA: 0s - loss: 0.3773 - accuracy: 0.8674
49/126 [==========>...................] - ETA: 0s - loss: 0.3820 - accuracy: 0.8699
60/126 [=============>................] - ETA: 0s - loss: 0.3773 - accuracy: 0.8672
72/126 [================>.............] - ETA: 0s - loss: 0.3877 - accuracy: 0.8672
84/126 [===================>..........] - ETA: 0s - loss: 0.3925 - accuracy: 0.8650
96/126 [=====================>........] - ETA: 0s - loss: 0.3944 - accuracy: 0.8620
108/126 [========================>.....] - ETA: 0s - loss: 0.3934 - accuracy: 0.8623
120/126 [===========================>..] - ETA: 0s - loss: 0.3980 - accuracy: 0.8628
126/126 [==============================] - 1s 4ms/step - loss: 0.3949 - accuracy: 0.8624
Epoch 7/10
1/126 [..............................] - ETA: 0s - loss: 0.2022 - accuracy: 0.9375
13/126 [==>...........................] - ETA: 0s - loss: 0.3838 - accuracy: 0.8774
25/126 [====>.........................] - ETA: 0s - loss: 0.3768 - accuracy: 0.8775
37/126 [=======>......................] - ETA: 0s - loss: 0.3513 - accuracy: 0.8801
49/126 [==========>...................] - ETA: 0s - loss: 0.3411 - accuracy: 0.8852
61/126 [=============>................] - ETA: 0s - loss: 0.3486 - accuracy: 0.8781
73/126 [================>.............] - ETA: 0s - loss: 0.3523 - accuracy: 0.8780
85/126 [===================>..........] - ETA: 0s - loss: 0.3631 - accuracy: 0.8757
97/126 [======================>.......] - ETA: 0s - loss: 0.3614 - accuracy: 0.8769
109/126 [========================>.....] - ETA: 0s - loss: 0.3655 - accuracy: 0.8761
121/126 [===========================>..] - ETA: 0s - loss: 0.3640 - accuracy: 0.8771
126/126 [==============================] - 1s 4ms/step - loss: 0.3635 - accuracy: 0.8774
Epoch 8/10
1/126 [..............................] - ETA: 0s - loss: 0.3186 - accuracy: 0.8750
13/126 [==>...........................] - ETA: 0s - loss: 0.3331 - accuracy: 0.8846
25/126 [====>.........................] - ETA: 0s - loss: 0.3351 - accuracy: 0.8850
37/126 [=======>......................] - ETA: 0s - loss: 0.3366 - accuracy: 0.8843
49/126 [==========>...................] - ETA: 0s - loss: 0.3372 - accuracy: 0.8852
61/126 [=============>................] - ETA: 0s - loss: 0.3334 - accuracy: 0.8842
73/126 [================>.............] - ETA: 0s - loss: 0.3354 - accuracy: 0.8848
85/126 [===================>..........] - ETA: 0s - loss: 0.3371 - accuracy: 0.8860
97/126 [======================>.......] - ETA: 0s - loss: 0.3355 - accuracy: 0.8860
109/126 [========================>.....] - ETA: 0s - loss: 0.3349 - accuracy: 0.8845
121/126 [===========================>..] - ETA: 0s - loss: 0.3427 - accuracy: 0.8830
126/126 [==============================] - 1s 4ms/step - loss: 0.3457 - accuracy: 0.8826
Epoch 9/10
1/126 [..............................] - ETA: 0s - loss: 0.2176 - accuracy: 0.9375
13/126 [==>...........................] - ETA: 0s - loss: 0.3388 - accuracy: 0.8822
25/126 [====>.........................] - ETA: 0s - loss: 0.3464 - accuracy: 0.8838
37/126 [=======>......................] - ETA: 0s - loss: 0.3463 - accuracy: 0.8742
49/126 [==========>...................] - ETA: 0s - loss: 0.3446 - accuracy: 0.8737
61/126 [=============>................] - ETA: 0s - loss: 0.3297 - accuracy: 0.8811
73/126 [================>.............] - ETA: 0s - loss: 0.3296 - accuracy: 0.8827
85/126 [===================>..........] - ETA: 0s - loss: 0.3323 - accuracy: 0.8816
97/126 [======================>.......] - ETA: 0s - loss: 0.3263 - accuracy: 0.8860
109/126 [========================>.....] - ETA: 0s - loss: 0.3284 - accuracy: 0.8873
121/126 [===========================>..] - ETA: 0s - loss: 0.3296 - accuracy: 0.8874
126/126 [==============================] - 1s 4ms/step - loss: 0.3285 - accuracy: 0.8878
Epoch 10/10
1/126 [..............................] - ETA: 0s - loss: 0.3079 - accuracy: 0.8125
13/126 [==>...........................] - ETA: 0s - loss: 0.2863 - accuracy: 0.9087
25/126 [====>.........................] - ETA: 0s - loss: 0.2799 - accuracy: 0.9075
37/126 [=======>......................] - ETA: 0s - loss: 0.2985 - accuracy: 0.9054
49/126 [==========>...................] - ETA: 0s - loss: 0.2956 - accuracy: 0.9037
67/126 [==============>...............] - ETA: 0s - loss: 0.2908 - accuracy: 0.9053
79/126 [=================>............] - ETA: 0s - loss: 0.2832 - accuracy: 0.9066
91/126 [====================>.........] - ETA: 0s - loss: 0.2922 - accuracy: 0.9052
103/126 [=======================>......] - ETA: 0s - loss: 0.2963 - accuracy: 0.9017
115/126 [==========================>...] - ETA: 0s - loss: 0.2983 - accuracy: 0.9000
126/126 [==============================] - 1s 4ms/step - loss: 0.3008 - accuracy: 0.8993
Datapoints: 0%| | 0/33 [00:00<?, ?it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 3%|▎ | 1/33 [00:00<00:08, 3.58it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 6%|▌ | 2/33 [00:00<00:08, 3.46it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 9%|▉ | 3/33 [00:00<00:08, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 12%|█▏ | 4/33 [00:01<00:08, 3.35it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 15%|█▌ | 5/33 [00:01<00:08, 3.36it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 18%|█▊ | 6/33 [00:01<00:08, 3.37it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 21%|██ | 7/33 [00:02<00:07, 3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 24%|██▍ | 8/33 [00:02<00:07, 3.46it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step
Datapoints: 27%|██▋ | 9/33 [00:02<00:06, 3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 30%|███ | 10/33 [00:02<00:06, 3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 33%|███▎ | 11/33 [00:03<00:06, 3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 36%|███▋ | 12/33 [00:03<00:06, 3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 39%|███▉ | 13/33 [00:03<00:05, 3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 42%|████▏ | 14/33 [00:04<00:05, 3.37it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 45%|████▌ | 15/33 [00:04<00:05, 3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 48%|████▊ | 16/33 [00:04<00:04, 3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 52%|█████▏ | 17/33 [00:04<00:04, 3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 2ms/step
Datapoints: 55%|█████▍ | 18/33 [00:05<00:04, 3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 58%|█████▊ | 19/33 [00:05<00:04, 3.43it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 61%|██████ | 20/33 [00:05<00:03, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 64%|██████▎ | 21/33 [00:06<00:03, 3.46it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 67%|██████▋ | 22/33 [00:06<00:03, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 70%|██████▉ | 23/33 [00:06<00:02, 3.41it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 73%|███████▎ | 24/33 [00:07<00:02, 3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 76%|███████▌ | 25/33 [00:07<00:02, 3.44it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 79%|███████▉ | 26/33 [00:07<00:02, 3.42it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 82%|████████▏ | 27/33 [00:07<00:01, 3.46it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 85%|████████▍ | 28/33 [00:08<00:01, 3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 88%|████████▊ | 29/33 [00:08<00:01, 3.39it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 91%|█████████ | 30/33 [00:08<00:00, 3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 94%|█████████▍| 31/33 [00:09<00:00, 3.40it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 97%|█████████▋| 32/33 [00:09<00:00, 3.38it/s]
1/2 [==============>...............] - ETA: 0s
2/2 [==============================] - 0s 3ms/step
Datapoints: 100%|██████████| 33/33 [00:09<00:00, 3.39it/s]
Datapoints: 100%|██████████| 33/33 [00:09<00:00, 3.41it/s]
CV Folds: 100%|██████████| 3/3 [01:02<00:00, 20.17s/it]
CV Folds: 100%|██████████| 3/3 [01:02<00:00, 20.78s/it]
We can now look at the results per group:
cv_results["test_single_accuracy"]
[[0.85, 0.8666666666666667, 0.9333333333333333, 0.8, 0.85, 0.85, 0.85, 0.8833333333333333, 0.85, 0.85, 0.8166666666666667, 0.8666666666666667, 0.9333333333333333, 0.8, 0.8833333333333333, 0.8, 0.8, 0.8833333333333333, 0.7833333333333333, 0.8166666666666667, 0.8, 0.85, 0.8, 0.95, 0.8, 0.8833333333333333, 0.8833333333333333, 0.85, 0.85, 0.7833333333333333, 0.75, 0.8666666666666667, 0.8333333333333334, 0.9333333333333333], [0.7833333333333333, 0.8166666666666667, 0.7666666666666667, 0.7833333333333333, 0.85, 0.8, 0.8, 0.8166666666666667, 0.8, 0.8333333333333334, 0.8333333333333334, 0.8, 0.8166666666666667, 0.85, 0.75, 0.7, 0.8333333333333334, 0.8166666666666667, 0.8666666666666667, 0.7666666666666667, 0.7833333333333333, 0.7833333333333333, 0.7666666666666667, 0.8833333333333333, 0.8333333333333334, 0.7833333333333333, 0.8, 0.85, 0.85, 0.8166666666666667, 0.7666666666666667, 0.8, 0.8833333333333333], [0.7666666666666667, 0.9166666666666666, 0.8, 0.8, 0.7833333333333333, 0.85, 0.8833333333333333, 0.85, 0.9, 0.8833333333333333, 0.9, 0.85, 0.8833333333333333, 0.8166666666666667, 0.8333333333333334, 0.8333333333333334, 0.85, 0.7833333333333333, 0.7166666666666667, 0.7333333333333333, 0.75, 0.7833333333333333, 0.85, 0.7666666666666667, 0.7833333333333333, 0.9, 0.8666666666666667, 0.8333333333333334, 0.8333333333333334, 0.8166666666666667, 0.85, 0.85, 0.9]]
And the overall accuracy as the average over all samples of all groups within a fold:
cv_results["test_per_sample__accuracy"]
array([0.84705882, 0.80858586, 0.83080808])
Total running time of the script: (1 minutes 9.074 seconds)
Estimated memory usage: 278 MB