Note
Go to the end to download the full example code
Tensorflow/Keras#
Note
This example requires the tensorflow package to be installed.
Theoretically, tpcp is framework agnostic and can be used with any framework. However, due to the way some frameworks handle their objects, some special handling internally is required. Hence, this example does not only serve as example on how to use tensorflow with tpcp, but also as a test case for these special cases.
When using tpcp with any machine learning framework, you either want to use a pretrained model with a normal pipeline or a train your own model as part of an Optimizable Pipeline. Here we show the second case, as it is more complex, and you are likely able to figure out the first case yourself.
This means, we are planning to perform the following steps:
Create a pipeline that creates and trains a model.
Allow the modification of model hyperparameters.
Run a simple cross-validation to demonstrate the functionality.
This example reimplements the basic MNIST example from the [tensorflow documentation](https://www.tensorflow.org/tutorials/keras/classification).
Some Notes#
In this example we show how to implement a Pipeline that uses tensorflow. You could implement an Algorithm in a similar way. This would actually be easier, as no specific handling of the input data would be required. For a pipeline, we need to create a custom Dataset class, as this is the expected input for a pipeline.
The Dataset#
We are using the normal fashion MNIST dataset for this example It consists of 60.000 images of 28x28 pixels, each with a label. We will ignore the typical train-test split, as we want to do our own cross-validation.
In addition, we will simulate an additional “index level”. In this (and most typical deep learning datasets), each datapoint is one vector for which we can make one prediction. In tpcp, we usually deal with datasets, where you might have multiple pieces of information for each datapoint. For example, one datapoint could be a patient, for which we have an entire time series of measurements. We will simulate this here, by creating the index of our dataset as 1000 groups each containing 60 images.
Other than that, the dataset is pretty standard.
Besides the create_index method, we only need to implement the input_as_array and labels_as_array methods that
allow us to easily access the data once we selected a single group.
from functools import lru_cache
import numpy as np
import pandas as pd
import tensorflow as tf
from tpcp import Dataset
tf.keras.utils.set_random_seed(812)
tf.config.experimental.enable_op_determinism()
@lru_cache(maxsize=1)
def get_fashion_mnist_data():
# Note: We throw train and test sets together, as we don't care about the official split here.
# We will create our own split later.
(train_images, train_labels), (test_images, test_labels) = (
tf.keras.datasets.fashion_mnist.load_data()
)
return np.array(list(train_images) + list(test_images)), list(
train_labels
) + list(test_labels)
class FashionMNIST(Dataset):
def input_as_array(self) -> np.ndarray:
self.assert_is_single(None, "input_as_array")
group_id = int(self.group_label.group_id)
images, _ = get_fashion_mnist_data()
return (
images[group_id * 60 : (group_id + 1) * 60].reshape((60, 28, 28))
/ 255
)
def labels_as_array(self) -> np.ndarray:
self.assert_is_single(None, "labels_as_array")
group_id = int(self.group_label.group_id)
_, labels = get_fashion_mnist_data()
return np.array(labels[group_id * 60 : (group_id + 1) * 60])
def create_index(self) -> pd.DataFrame:
# There are 60.000 images in total.
# We simulate 1000 groups of 60 images each.
return pd.DataFrame({"group_id": list(range(1000))})
We can see our Dataset works as expected:
dataset = FashionMNIST()
dataset[0].input_as_array().shape
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz
0/29515 ━━━━━━━━━━━━━━━━━━━━ 0s 0s/step
29515/29515 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz
0/26421880 ━━━━━━━━━━━━━━━━━━━━ 0s 0s/step
81920/26421880 ━━━━━━━━━━━━━━━━━━━━ 20s 1us/step
344064/26421880 ━━━━━━━━━━━━━━━━━━━━ 8s 0us/step
868352/26421880 ━━━━━━━━━━━━━━━━━━━━ 4s 0us/step
1998848/26421880 ━━━━━━━━━━━━━━━━━━━━ 2s 0us/step
4767744/26421880 ━━━━━━━━━━━━━━━━━━━━ 1s 0us/step
10526720/26421880 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
17874944/26421880 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
26345472/26421880 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
26421880/26421880 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz
0/5148 ━━━━━━━━━━━━━━━━━━━━ 0s 0s/step
5148/5148 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz
0/4422102 ━━━━━━━━━━━━━━━━━━━━ 0s 0s/step
401408/4422102 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
2105344/4422102 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
4422102/4422102 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step
(60, 28, 28)
dataset[0].labels_as_array().shape
(60,)
The Pipeline#
We will create a pipeline that uses a simple neural network to classify the images.
In tpcp, all “things” that should be optimized need to be parameters.
This means our model itself needs to be a parameter of the pipeline.
However, as we don’t have the model yet, as its creation depends on other hyperparameters, we add it as an optional
parameter initialized with None.
Further, we prefix the parameter name with an underscore, to signify, that this is not a parameter that should be
modified manually by the user.
This is just convention, and it is up to you to decide how you want to name your parameters.
We further introduce a hyperparameter n_dense_layer_nodes to show how we can influence the model creation.
The optimize method#
To make our pipeline optimizable, it needs to inherit from OptimizablePipeline.
Further we need to mark at least one of the parameters as OptiPara using the type annotation.
We do this for our _model parameter.
Finally, we need to implement the self_optimize method.
This method will get the entire training dataset as input and should update the _model parameter with the trained
model.
Hence, we first extract the relevant data (remember, each datapoint is 60 images), by concatinating all images over
all groups in the dataset.
Then we create the Keras model based on the hyperparameters.
Finally, we train the model and update the _model parameter.
Here we chose to wrap the method with make_optimize_safe.
This decorator will perform some runtime checks to ensure that the method is implemented correctly.
The run method#
The run method expects that the _model parameter is already set (i.e. the pipeline was already optimized).
It gets a single datapoint as input (remember, a datapoint is a single group of 60 images).
We then extract the data from the datapoint and let the model make a prediction.
We store the prediction on our output attribute predictions_.
The trailing underscore is a convention to signify, that this is an “result” attribute.
import warnings
from typing import Optional
from tpcp import (
OptimizablePipeline,
OptiPara,
make_action_safe,
make_optimize_safe,
)
from typing_extensions import Self
class KerasPipeline(OptimizablePipeline):
n_dense_layer_nodes: int
n_train_epochs: int
_model: OptiPara[Optional[tf.keras.Sequential]]
predictions_: np.ndarray
def __init__(
self,
n_dense_layer_nodes=128,
n_train_epochs=5,
_model: Optional[tf.keras.Sequential] = None,
):
self.n_dense_layer_nodes = n_dense_layer_nodes
self.n_train_epochs = n_train_epochs
self._model = _model
@property
def predicted_labels_(self):
return np.argmax(self.predictions_, axis=1)
@make_optimize_safe
def self_optimize(self, dataset, **_) -> Self:
data = tf.convert_to_tensor(
np.vstack([d.input_as_array() for d in dataset])
)
labels = tf.convert_to_tensor(
np.hstack([d.labels_as_array() for d in dataset])
)
print(data.shape)
if self._model is not None:
warnings.warn("Overwriting existing model!")
self._model = tf.keras.Sequential(
[
tf.keras.layers.Input((28, 28)),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(
self.n_dense_layer_nodes, activation="relu"
),
tf.keras.layers.Dense(10),
]
)
self._model.compile(
optimizer="adam",
loss=tf.keras.losses.SparseCategoricalCrossentropy(
from_logits=True
),
metrics=["accuracy"],
)
self._model.fit(data, labels, epochs=self.n_train_epochs)
return self
@make_action_safe
def run(self, datapoint) -> Self:
if self._model is None:
raise RuntimeError("Model not trained yet!")
data = tf.convert_to_tensor(datapoint.input_as_array())
self.predictions_ = self._model.predict(data)
return self
Testing the pipeline#
We can now test our pipeline.
We will run the optimization using a couple of datapoints (to keep everything fast) and then use run to get the
predictions for a single unseen datapoint.
pipeline = KerasPipeline().self_optimize(FashionMNIST()[:10])
p1 = pipeline.run(FashionMNIST()[11])
print(p1.predicted_labels_)
print(FashionMNIST()[11].labels_as_array())
(600, 28, 28)
Epoch 1/5
1/19 ━━━━━━━━━━━━━━━━━━━━ 11s 638ms/step - accuracy: 0.1250 - loss: 2.3593
19/19 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.4933 - loss: 1.5112
Epoch 2/5
1/19 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step - accuracy: 0.7188 - loss: 1.0003
19/19 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7167 - loss: 0.8663
Epoch 3/5
1/19 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step - accuracy: 0.7812 - loss: 0.7427
19/19 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7800 - loss: 0.6830
Epoch 4/5
1/19 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step - accuracy: 0.7812 - loss: 0.6089
19/19 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8133 - loss: 0.5658
Epoch 5/5
1/19 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step - accuracy: 0.7812 - loss: 0.5329
19/19 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8433 - loss: 0.4940
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
[8 8 6 9 4 0 7 3 7 9 4 8 4 3 7 8 1 4 0 7 9 8 5 5 2 1 3 4 1 9 7 5 9 9 7 8 2
7 4 7 2 4 7 1 1 7 5 4 8 3 5 9 0 7 4 0 0 9 1 9]
[8 8 0 9 2 0 7 3 7 9 3 8 4 3 7 8 1 4 0 7 9 8 5 5 2 1 3 4 6 7 7 5 9 9 7 8 2
7 4 7 0 3 5 1 1 5 5 2 8 3 5 9 0 7 3 0 0 7 1 9]
We can see that even with just 5 epochs, the model already performs quite well. To quantify we can calculate the accuracy for this datapoint:
from sklearn.metrics import accuracy_score
accuracy_score(p1.predicted_labels_, FashionMNIST()[11].labels_as_array())
0.8
Cross Validation#
If we want to run a cross validation, we need to formalize the scoring into a function. We will calculate two types of accuracy: First, the accuracy per group and second, the accuracy over all images across all groups. For more information about how this works, check the Custom Scorer example.
from collections.abc import Sequence
from tpcp.validate import Aggregator
class SingleValueAccuracy(Aggregator[tuple[np.ndarray, np.ndarray]]):
def aggregate(
self, /, values: Sequence[tuple[np.ndarray, np.ndarray]], **_
) -> dict[str, float]:
return {
"accuracy": accuracy_score(
np.hstack([v[0] for v in values]),
np.hstack([v[1] for v in values]),
)
}
single_value_accuracy = SingleValueAccuracy()
def scoring(pipeline, datapoint):
result: np.ndarray = pipeline.safe_run(datapoint).predicted_labels_
reference = datapoint.labels_as_array()
return {
"accuracy": accuracy_score(result, reference),
"per_sample": single_value_accuracy((result, reference)),
}
Now we can run a cross validation. We will only run it on a subset of the data, to keep the runtime manageable.
Note
You might see warnings about retracing of the model. This is because we clone the pipeline before each call to the run method. This is a good idea to ensure that all pipelines are independent of each other, however, might result in some performance overhead.
from tpcp.optimize import Optimize
from tpcp.validate import cross_validate
pipeline = KerasPipeline(n_train_epochs=10)
cv_results = cross_validate(
Optimize(pipeline), FashionMNIST()[:100], scoring=scoring, cv=3
)
CV Folds: 0%| | 0/3 [00:00<?, ?it/s](3960, 28, 28)
Epoch 1/10
1/124 ━━━━━━━━━━━━━━━━━━━━ 1:16 623ms/step - accuracy: 0.0938 - loss: 2.5074
25/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.3223 - loss: 1.8317
51/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.4313 - loss: 1.5513
77/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.4946 - loss: 1.3891
103/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.5352 - loss: 1.2854
124/124 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.6919 - loss: 0.8968
Epoch 2/10
1/124 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8750 - loss: 0.4848
27/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7852 - loss: 0.6248
54/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7905 - loss: 0.6066
80/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7946 - loss: 0.5944
107/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7972 - loss: 0.5890
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8111 - loss: 0.5677
Epoch 3/10
1/124 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8438 - loss: 0.4153
27/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8130 - loss: 0.5246
53/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8224 - loss: 0.5109
79/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8267 - loss: 0.5031
105/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8283 - loss: 0.5010
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8351 - loss: 0.4936
Epoch 4/10
1/124 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8750 - loss: 0.3588
26/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8301 - loss: 0.4582
52/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8374 - loss: 0.4516
78/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8399 - loss: 0.4482
105/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8411 - loss: 0.4487
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8482 - loss: 0.4475
Epoch 5/10
1/124 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8750 - loss: 0.3387
27/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8460 - loss: 0.4158
54/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8514 - loss: 0.4121
81/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8534 - loss: 0.4105
107/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8548 - loss: 0.4116
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8614 - loss: 0.4122
Epoch 6/10
1/124 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8750 - loss: 0.3044
27/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8592 - loss: 0.3745
54/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8628 - loss: 0.3759
80/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8629 - loss: 0.3774
106/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8633 - loss: 0.3799
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8674 - loss: 0.3840
Epoch 7/10
1/124 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.9062 - loss: 0.2879
27/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8611 - loss: 0.3449
53/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8656 - loss: 0.3481
79/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8672 - loss: 0.3508
105/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8683 - loss: 0.3533
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8742 - loss: 0.3577
Epoch 8/10
1/124 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8750 - loss: 0.2663
27/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8681 - loss: 0.3192
53/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8734 - loss: 0.3242
80/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8759 - loss: 0.3280
107/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8773 - loss: 0.3308
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8828 - loss: 0.3357
Epoch 9/10
1/124 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.9375 - loss: 0.2468
28/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8888 - loss: 0.2965
54/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8883 - loss: 0.3017
81/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8880 - loss: 0.3054
108/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8880 - loss: 0.3085
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8902 - loss: 0.3147
Epoch 10/10
1/124 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.9375 - loss: 0.2620
28/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8990 - loss: 0.2788
54/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8967 - loss: 0.2844
81/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8959 - loss: 0.2879
108/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8951 - loss: 0.2907
124/124 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8947 - loss: 0.2970
Datapoints: 0%| | 0/34 [00:00<?, ?it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 3%|▎ | 1/34 [00:00<00:04, 8.06it/s]WARNING:tensorflow:5 out of the last 5 calls to <function TensorFlowTrainer.make_predict_function.<locals>.one_step_on_data_distributed at 0x74d7c4427c70> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/stepWARNING:tensorflow:6 out of the last 6 calls to <function TensorFlowTrainer.make_predict_function.<locals>.one_step_on_data_distributed at 0x74d7c4427c70> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 6%|▌ | 2/34 [00:00<00:04, 7.88it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 9%|▉ | 3/34 [00:00<00:03, 7.99it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 12%|█▏ | 4/34 [00:00<00:03, 8.06it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 15%|█▍ | 5/34 [00:00<00:03, 7.96it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 18%|█▊ | 6/34 [00:00<00:03, 7.96it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 21%|██ | 7/34 [00:00<00:03, 7.98it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 24%|██▎ | 8/34 [00:01<00:03, 7.91it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 26%|██▋ | 9/34 [00:01<00:03, 7.94it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 29%|██▉ | 10/34 [00:01<00:03, 8.00it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 32%|███▏ | 11/34 [00:01<00:02, 7.96it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 35%|███▌ | 12/34 [00:01<00:02, 7.99it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 38%|███▊ | 13/34 [00:01<00:02, 8.03it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 41%|████ | 14/34 [00:01<00:02, 8.00it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 44%|████▍ | 15/34 [00:01<00:02, 7.98it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 47%|████▋ | 16/34 [00:02<00:02, 8.01it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 50%|█████ | 17/34 [00:02<00:02, 7.94it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 53%|█████▎ | 18/34 [00:02<00:02, 7.96it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 56%|█████▌ | 19/34 [00:02<00:01, 7.97it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 59%|█████▉ | 20/34 [00:02<00:01, 7.94it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 33ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 62%|██████▏ | 21/34 [00:02<00:01, 7.95it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 33ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 65%|██████▍ | 22/34 [00:02<00:01, 7.95it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 34ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
Datapoints: 68%|██████▊ | 23/34 [00:02<00:01, 7.80it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 71%|███████ | 24/34 [00:03<00:01, 7.84it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 74%|███████▎ | 25/34 [00:03<00:01, 7.90it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 76%|███████▋ | 26/34 [00:03<00:01, 7.88it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 79%|███████▉ | 27/34 [00:03<00:00, 7.94it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 82%|████████▏ | 28/34 [00:03<00:00, 7.98it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 85%|████████▌ | 29/34 [00:03<00:00, 7.92it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 88%|████████▊ | 30/34 [00:03<00:00, 7.97it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 91%|█████████ | 31/34 [00:03<00:00, 7.99it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 33ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
Datapoints: 94%|█████████▍| 32/34 [00:04<00:00, 7.84it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 97%|█████████▋| 33/34 [00:04<00:00, 7.84it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 100%|██████████| 34/34 [00:04<00:00, 7.89it/s]
Datapoints: 100%|██████████| 34/34 [00:04<00:00, 7.94it/s]
CV Folds: 33%|███▎ | 1/3 [00:08<00:16, 8.19s/it](4020, 28, 28)
Epoch 1/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1:55 923ms/step - accuracy: 0.1250 - loss: 2.3656
26/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.3664 - loss: 1.8220
53/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.4718 - loss: 1.5464
79/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.5233 - loss: 1.3997
106/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.5603 - loss: 1.2939
126/126 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7000 - loss: 0.8913
Epoch 2/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 2s 20ms/step - accuracy: 0.8438 - loss: 0.4830
25/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7650 - loss: 0.6233
51/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7666 - loss: 0.6231
77/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7712 - loss: 0.6176
104/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7756 - loss: 0.6111
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7968 - loss: 0.5690
Epoch 3/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8750 - loss: 0.3561
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8125 - loss: 0.5183
54/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8123 - loss: 0.5234
80/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8144 - loss: 0.5208
107/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8163 - loss: 0.5178
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8284 - loss: 0.4927
Epoch 4/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.9375 - loss: 0.3082
19/126 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8418 - loss: 0.4642
36/126 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8360 - loss: 0.4791
61/126 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8335 - loss: 0.4831
87/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8346 - loss: 0.4806
114/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8357 - loss: 0.4774
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8453 - loss: 0.4533
Epoch 5/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8750 - loss: 0.3008
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8343 - loss: 0.4349
54/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8369 - loss: 0.4398
81/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8403 - loss: 0.4362
107/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8427 - loss: 0.4341
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8570 - loss: 0.4134
Epoch 6/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8750 - loss: 0.2896
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8400 - loss: 0.4038
53/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8444 - loss: 0.4079
80/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8497 - loss: 0.4044
106/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8526 - loss: 0.4025
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8674 - loss: 0.3836
Epoch 7/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8750 - loss: 0.2921
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8356 - loss: 0.3781
54/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8426 - loss: 0.3813
81/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8496 - loss: 0.3775
108/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8536 - loss: 0.3756
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8724 - loss: 0.3583
Epoch 8/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 14ms/step - accuracy: 0.8750 - loss: 0.2784
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8482 - loss: 0.3537
54/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8538 - loss: 0.3574
81/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8597 - loss: 0.3543
108/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8633 - loss: 0.3527
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8801 - loss: 0.3375
Epoch 9/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8750 - loss: 0.2839
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8577 - loss: 0.3366
52/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8619 - loss: 0.3380
79/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8673 - loss: 0.3340
106/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8710 - loss: 0.3323
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8873 - loss: 0.3178
Epoch 10/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8750 - loss: 0.2753
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8626 - loss: 0.3165
53/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8687 - loss: 0.3180
79/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8747 - loss: 0.3144
106/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8785 - loss: 0.3129
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8940 - loss: 0.3003
Datapoints: 0%| | 0/33 [00:00<?, ?it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 3%|▎ | 1/33 [00:00<00:03, 8.06it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 6%|▌ | 2/33 [00:00<00:03, 7.89it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 9%|▉ | 3/33 [00:00<00:03, 7.97it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 12%|█▏ | 4/33 [00:00<00:03, 8.03it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 15%|█▌ | 5/33 [00:00<00:03, 7.97it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 18%|█▊ | 6/33 [00:00<00:03, 8.02it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 21%|██ | 7/33 [00:00<00:03, 8.07it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 34ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 24%|██▍ | 8/33 [00:01<00:03, 7.98it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 33ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 27%|██▋ | 9/33 [00:01<00:03, 7.99it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 30%|███ | 10/33 [00:01<00:02, 8.03it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 34ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 33%|███▎ | 11/33 [00:01<00:02, 8.01it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 36%|███▋ | 12/33 [00:01<00:02, 8.07it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 39%|███▉ | 13/33 [00:01<00:02, 8.03it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 34ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 42%|████▏ | 14/33 [00:01<00:02, 8.00it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 45%|████▌ | 15/33 [00:01<00:02, 8.04it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 48%|████▊ | 16/33 [00:01<00:02, 8.05it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 36ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 52%|█████▏ | 17/33 [00:02<00:02, 7.97it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 55%|█████▍ | 18/33 [00:02<00:01, 8.00it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
Datapoints: 58%|█████▊ | 19/33 [00:02<00:01, 7.98it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 61%|██████ | 20/33 [00:02<00:01, 7.93it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 64%|██████▎ | 21/33 [00:02<00:01, 7.98it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 67%|██████▋ | 22/33 [00:02<00:01, 8.02it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 70%|██████▉ | 23/33 [00:02<00:01, 7.97it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 73%|███████▎ | 24/33 [00:02<00:01, 8.02it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 76%|███████▌ | 25/33 [00:03<00:00, 8.07it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 36ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 79%|███████▉ | 26/33 [00:03<00:00, 7.91it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 30ms/step
Datapoints: 82%|████████▏ | 27/33 [00:03<00:00, 7.89it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 85%|████████▍ | 28/33 [00:03<00:00, 7.96it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
Datapoints: 88%|████████▊ | 29/33 [00:03<00:00, 7.94it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 91%|█████████ | 30/33 [00:03<00:00, 8.01it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 94%|█████████▍| 31/33 [00:03<00:00, 8.05it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 34ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 97%|█████████▋| 32/33 [00:04<00:00, 7.99it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 33ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 100%|██████████| 33/33 [00:04<00:00, 7.98it/s]
Datapoints: 100%|██████████| 33/33 [00:04<00:00, 7.99it/s]
CV Folds: 67%|██████▋ | 2/3 [00:16<00:08, 8.29s/it](4020, 28, 28)
Epoch 1/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1:13 589ms/step - accuracy: 0.0938 - loss: 2.3129
26/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.3843 - loss: 1.7676
52/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.4739 - loss: 1.5276
79/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.5263 - loss: 1.3824
106/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.5622 - loss: 1.2816
126/126 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.6983 - loss: 0.8965
Epoch 2/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8125 - loss: 0.4789
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7854 - loss: 0.6087
54/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7836 - loss: 0.6135
80/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7860 - loss: 0.6111
107/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7882 - loss: 0.6067
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8005 - loss: 0.5778
Epoch 3/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.8750 - loss: 0.3564
28/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8293 - loss: 0.5023
54/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8227 - loss: 0.5138
80/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8220 - loss: 0.5152
106/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8224 - loss: 0.5145
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8269 - loss: 0.5017
Epoch 4/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.9062 - loss: 0.3024
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8540 - loss: 0.4405
53/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8464 - loss: 0.4545
80/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8449 - loss: 0.4572
106/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8443 - loss: 0.4581
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8458 - loss: 0.4515
Epoch 5/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.9062 - loss: 0.2785
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8679 - loss: 0.4030
53/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8612 - loss: 0.4169
79/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8597 - loss: 0.4197
105/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8587 - loss: 0.4207
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8585 - loss: 0.4165
Epoch 6/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.9062 - loss: 0.2679
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8796 - loss: 0.3663
54/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8743 - loss: 0.3802
80/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8729 - loss: 0.3841
107/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8718 - loss: 0.3861
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8719 - loss: 0.3833
Epoch 7/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.9375 - loss: 0.2306
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8884 - loss: 0.3390
54/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8834 - loss: 0.3513
80/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8824 - loss: 0.3553
107/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8816 - loss: 0.3583
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8823 - loss: 0.3594
Epoch 8/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.9375 - loss: 0.2117
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8936 - loss: 0.3167
54/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8897 - loss: 0.3291
81/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8885 - loss: 0.3337
109/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8878 - loss: 0.3361
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8886 - loss: 0.3366
Epoch 9/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.9375 - loss: 0.2051
27/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8974 - loss: 0.2982
53/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8944 - loss: 0.3090
79/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8934 - loss: 0.3127
105/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8938 - loss: 0.3148
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8973 - loss: 0.3152
Epoch 10/10
1/126 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - accuracy: 0.9375 - loss: 0.1920
28/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8987 - loss: 0.2813
55/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8979 - loss: 0.2916
82/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8979 - loss: 0.2961
109/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8983 - loss: 0.2985
126/126 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9030 - loss: 0.3000
Datapoints: 0%| | 0/33 [00:00<?, ?it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 3%|▎ | 1/33 [00:00<00:03, 8.13it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 6%|▌ | 2/33 [00:00<00:03, 8.05it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 9%|▉ | 3/33 [00:00<00:03, 8.11it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 12%|█▏ | 4/33 [00:00<00:03, 8.10it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 15%|█▌ | 5/33 [00:00<00:03, 8.02it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 18%|█▊ | 6/33 [00:00<00:03, 8.08it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 21%|██ | 7/33 [00:00<00:03, 8.13it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 24%|██▍ | 8/33 [00:00<00:03, 8.10it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 27%|██▋ | 9/33 [00:01<00:02, 8.13it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 30%|███ | 10/33 [00:01<00:02, 8.08it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 33%|███▎ | 11/33 [00:01<00:02, 8.00it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 36%|███▋ | 12/33 [00:01<00:02, 7.98it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 39%|███▉ | 13/33 [00:01<00:02, 8.04it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 42%|████▏ | 14/33 [00:01<00:02, 8.02it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 45%|████▌ | 15/33 [00:01<00:02, 8.06it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 48%|████▊ | 16/33 [00:01<00:02, 8.11it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 52%|█████▏ | 17/33 [00:02<00:01, 8.03it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 55%|█████▍ | 18/33 [00:02<00:01, 8.05it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 58%|█████▊ | 19/33 [00:02<00:01, 8.07it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 61%|██████ | 20/33 [00:02<00:01, 7.97it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 64%|██████▎ | 21/33 [00:02<00:01, 7.99it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 67%|██████▋ | 22/33 [00:02<00:01, 8.04it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 33ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 70%|██████▉ | 23/33 [00:02<00:01, 7.96it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 73%|███████▎ | 24/33 [00:02<00:01, 8.00it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 76%|███████▌ | 25/33 [00:03<00:00, 8.04it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 79%|███████▉ | 26/33 [00:03<00:00, 7.96it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 82%|████████▏ | 27/33 [00:03<00:00, 7.99it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 85%|████████▍ | 28/33 [00:03<00:00, 8.03it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 88%|████████▊ | 29/33 [00:03<00:00, 7.98it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step
Datapoints: 91%|█████████ | 30/33 [00:03<00:00, 8.05it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 33ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 94%|█████████▍| 31/33 [00:03<00:00, 8.06it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step
Datapoints: 97%|█████████▋| 32/33 [00:03<00:00, 7.94it/s]
1/2 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 30ms/step
Datapoints: 100%|██████████| 33/33 [00:04<00:00, 7.95it/s]
Datapoints: 100%|██████████| 33/33 [00:04<00:00, 8.03it/s]
CV Folds: 100%|██████████| 3/3 [00:24<00:00, 8.15s/it]
CV Folds: 100%|██████████| 3/3 [00:24<00:00, 8.18s/it]
We can now look at the results per group:
cv_results["test__single__accuracy"]
[[0.8333333333333334, 0.8333333333333334, 0.9166666666666666, 0.7666666666666667, 0.8166666666666667, 0.8166666666666667, 0.8, 0.8333333333333334, 0.8333333333333334, 0.85, 0.7666666666666667, 0.8333333333333334, 0.8833333333333333, 0.8666666666666667, 0.8833333333333333, 0.8666666666666667, 0.8333333333333334, 0.8333333333333334, 0.8, 0.8333333333333334, 0.7833333333333333, 0.7333333333333333, 0.8, 0.8333333333333334, 0.8333333333333334, 0.8833333333333333, 0.8666666666666667, 0.85, 0.85, 0.75, 0.75, 0.85, 0.8166666666666667, 0.8833333333333333], [0.8, 0.9, 0.75, 0.8666666666666667, 0.8833333333333333, 0.8166666666666667, 0.8833333333333333, 0.8833333333333333, 0.8333333333333334, 0.8, 0.85, 0.8, 0.8166666666666667, 0.8, 0.7833333333333333, 0.8, 0.8166666666666667, 0.7666666666666667, 0.85, 0.75, 0.8, 0.8333333333333334, 0.7666666666666667, 0.9, 0.8, 0.7833333333333333, 0.8166666666666667, 0.8666666666666667, 0.8833333333333333, 0.9166666666666666, 0.7333333333333333, 0.7833333333333333, 0.9], [0.7833333333333333, 0.9333333333333333, 0.75, 0.8333333333333334, 0.8833333333333333, 0.8666666666666667, 0.8833333333333333, 0.9, 0.9, 0.9, 0.9166666666666666, 0.85, 0.8666666666666667, 0.8833333333333333, 0.85, 0.8, 0.9166666666666666, 0.75, 0.7166666666666667, 0.8666666666666667, 0.8166666666666667, 0.75, 0.7833333333333333, 0.7333333333333333, 0.8666666666666667, 0.8833333333333333, 0.9166666666666666, 0.8, 0.8666666666666667, 0.9, 0.85, 0.85, 0.9]]
Average first per group and then over all groups:
cv_results["test__agg__accuracy"]
array([0.82892157, 0.82525253, 0.84747475])
And the overall accuracy as the average over all samples of all groups within a fold:
cv_results["test__agg__per_sample__accuracy"]
array([0.82892157, 0.82525253, 0.84747475])
Total running time of the script: (0 minutes 38.775 seconds)
Estimated memory usage: 843 MB