.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/integrations/_01_tensorflow.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_integrations__01_tensorflow.py: .. _tensorflow: Tensorflow/Keras ================ .. note:: This example requires the `tensorflow` package to be installed. Theoretically, tpcp is framework agnostic and can be used with any framework. However, due to the way some frameworks handle their objects, some special handling internally is required. Hence, this example does not only serve as example on how to use tensorflow with tpcp, but also as a test case for these special cases. When using tpcp with any machine learning framework, you either want to use a pretrained model with a normal pipeline or a train your own model as part of an Optimizable Pipeline. Here we show the second case, as it is more complex, and you are likely able to figure out the first case yourself. This means, we are planning to perform the following steps: 1. Create a pipeline that creates and trains a model. 2. Allow the modification of model hyperparameters. 3. Run a simple cross-validation to demonstrate the functionality. This example reimplements the basic MNIST example from the [tensorflow documentation](https://www.tensorflow.org/tutorials/keras/classification). Some Notes ---------- In this example we show how to implement a Pipeline that uses tensorflow. You could implement an Algorithm in a similar way. This would actually be easier, as no specific handling of the input data would be required. For a pipeline, we need to create a custom Dataset class, as this is the expected input for a pipeline. .. GENERATED FROM PYTHON SOURCE LINES 39-54 The Dataset ----------- We are using the normal fashion MNIST dataset for this example It consists of 60.000 images of 28x28 pixels, each with a label. We will ignore the typical train-test split, as we want to do our own cross-validation. In addition, we will simulate an additional "index level". In this (and most typical deep learning datasets), each datapoint is one vector for which we can make one prediction. In tpcp, we usually deal with datasets, where you might have multiple pieces of information for each datapoint. For example, one datapoint could be a patient, for which we have an entire time series of measurements. We will simulate this here, by creating the index of our dataset as 1000 groups each containing 60 images. Other than that, the dataset is pretty standard. Besides the `create_index` method, we only need to implement the `input_as_array` and `labels_as_array` methods that allow us to easily access the data once we selected a single group. .. GENERATED FROM PYTHON SOURCE LINES 54-93 .. code-block:: default from functools import lru_cache import numpy as np import pandas as pd import tensorflow as tf from tpcp import Dataset tf.keras.utils.set_random_seed(812) tf.config.experimental.enable_op_determinism() @lru_cache(maxsize=1) def get_fashion_mnist_data(): # Note: We throw train and test sets together, as we don't care about the official split here. # We will create our own split later. (train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.fashion_mnist.load_data() return np.array(list(train_images) + list(test_images)), list(train_labels) + list(test_labels) class FashionMNIST(Dataset): def input_as_array(self) -> np.ndarray: self.assert_is_single(None, "input_as_array") group_id = int(self.group_label.group_id) images, _ = get_fashion_mnist_data() return images[group_id * 60 : (group_id + 1) * 60].reshape((60, 28, 28)) / 255 def labels_as_array(self) -> np.ndarray: self.assert_is_single(None, "labels_as_array") group_id = int(self.group_label.group_id) _, labels = get_fashion_mnist_data() return np.array(labels[group_id * 60 : (group_id + 1) * 60]) def create_index(self) -> pd.DataFrame: # There are 60.000 images in total. # We simulate 1000 groups of 60 images each. return pd.DataFrame({"group_id": list(range(1000))}) .. GENERATED FROM PYTHON SOURCE LINES 94-95 We can see our Dataset works as expected: .. GENERATED FROM PYTHON SOURCE LINES 95-98 .. code-block:: default dataset = FashionMNIST() dataset[0].input_as_array().shape .. rst-class:: sphx-glr-script-out .. code-block:: none Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz 8192/29515 [=======>......................] - ETA: 0s 29515/29515 [==============================] - 0s 1us/step Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz 8192/26421880 [..............................] - ETA: 0s 122880/26421880 [..............................] - ETA: 10s 909312/26421880 [>.............................] - ETA: 2s  5046272/26421880 [====>.........................] - ETA: 0s 9904128/26421880 [==========>...................] - ETA: 0s 14852096/26421880 [===============>..............] - ETA: 0s 19505152/26421880 [=====================>........] - ETA: 0s 24387584/26421880 [==========================>...] - ETA: 0s 26421880/26421880 [==============================] - 0s 0us/step Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz 5148/5148 [==============================] - 0s 0us/step Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz 8192/4422102 [..............................] - ETA: 0s 131072/4422102 [..............................] - ETA: 1s 671744/4422102 [===>..........................] - ETA: 0s 3997696/4422102 [==========================>...] - ETA: 0s 4422102/4422102 [==============================] - 0s 0us/step (60, 28, 28) .. GENERATED FROM PYTHON SOURCE LINES 99-101 .. code-block:: default dataset[0].labels_as_array().shape .. rst-class:: sphx-glr-script-out .. code-block:: none (60,) .. GENERATED FROM PYTHON SOURCE LINES 102-139 The Pipeline ------------ We will create a pipeline that uses a simple neural network to classify the images. In tpcp, all "things" that should be optimized need to be parameters. This means our model itself needs to be a parameter of the pipeline. However, as we don't have the model yet, as its creation depends on other hyperparameters, we add it as an optional parameter initialized with `None`. Further, we prefix the parameter name with an underscore, to signify, that this is not a parameter that should be modified manually by the user. This is just convention, and it is up to you to decide how you want to name your parameters. We further introduce a hyperparameter `n_dense_layer_nodes` to show how we can influence the model creation. The optimize method +++++++++++++++++++ To make our pipeline optimizable, it needs to inherit from `OptimizablePipeline`. Further we need to mark at least one of the parameters as `OptiPara` using the type annotation. We do this for our `_model` parameter. Finally, we need to implement the `self_optimize` method. This method will get the entire training dataset as input and should update the `_model` parameter with the trained model. Hence, we first extract the relevant data (remember, each datapoint is 60 images), by concatinating all images over all groups in the dataset. Then we create the Keras model based on the hyperparameters. Finally, we train the model and update the `_model` parameter. Here we chose to wrap the method with `make_optimize_safe`. This decorator will perform some runtime checks to ensure that the method is implemented correctly. The run method ++++++++++++++ The run method expects that the `_model` parameter is already set (i.e. the pipeline was already optimized). It gets a single datapoint as input (remember, a datapoint is a single group of 60 images). We then extract the data from the datapoint and let the model make a prediction. We store the prediction on our output attribute `predictions_`. The trailing underscore is a convention to signify, that this is an "result" attribute. .. GENERATED FROM PYTHON SOURCE LINES 139-200 .. code-block:: default import warnings from typing import Optional from typing_extensions import Self from tpcp import OptimizablePipeline, OptiPara, make_action_safe, make_optimize_safe class KerasPipeline(OptimizablePipeline): n_dense_layer_nodes: int n_train_epochs: int _model: OptiPara[Optional[tf.keras.Sequential]] predictions_: np.ndarray def __init__(self, n_dense_layer_nodes=128, n_train_epochs=5, _model: Optional[tf.keras.Sequential] = None): self.n_dense_layer_nodes = n_dense_layer_nodes self.n_train_epochs = n_train_epochs self._model = _model @property def predicted_labels_(self): return np.argmax(self.predictions_, axis=1) @make_optimize_safe def self_optimize(self, dataset, **_) -> Self: data = np.vstack([d.input_as_array() for d in dataset]) labels = np.hstack([d.labels_as_array() for d in dataset]) print(data.shape) if self._model is not None: warnings.warn("Overwriting existing model!") self._model = tf.keras.Sequential( [ tf.keras.layers.Flatten(input_shape=(28, 28)), tf.keras.layers.Dense(self.n_dense_layer_nodes, activation="relu"), tf.keras.layers.Dense(10), ] ) self._model.compile( optimizer="adam", loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=["accuracy"], ) self._model.fit(data, labels, epochs=self.n_train_epochs) return self @make_action_safe def run(self, datapoint) -> Self: if self._model is None: raise RuntimeError("Model not trained yet!") data = datapoint.input_as_array() self.predictions_ = self._model.predict(data) return self .. GENERATED FROM PYTHON SOURCE LINES 201-206 Testing the pipeline -------------------- We can now test our pipeline. We will run the optimization using a couple of datapoints (to keep everything fast) and then use `run` to get the predictions for a single unseen datapoint. .. GENERATED FROM PYTHON SOURCE LINES 206-211 .. code-block:: default pipeline = KerasPipeline().self_optimize(FashionMNIST()[:10]) p1 = pipeline.run(FashionMNIST()[11]) print(p1.predicted_labels_) print(FashionMNIST()[11].labels_as_array()) .. rst-class:: sphx-glr-script-out .. code-block:: none (600, 28, 28) Epoch 1/5 1/19 [>.............................] - ETA: 19s - loss: 2.3593 - accuracy: 0.1250 13/19 [===================>..........] - ETA: 0s - loss: 1.6625 - accuracy: 0.4471  19/19 [==============================] - 1s 5ms/step - loss: 1.5112 - accuracy: 0.4933 Epoch 2/5 1/19 [>.............................] - ETA: 0s - loss: 0.9220 - accuracy: 0.7188 13/19 [===================>..........] - ETA: 0s - loss: 0.9038 - accuracy: 0.7115 19/19 [==============================] - 0s 5ms/step - loss: 0.8705 - accuracy: 0.7083 Epoch 3/5 1/19 [>.............................] - ETA: 0s - loss: 1.0179 - accuracy: 0.6875 13/19 [===================>..........] - ETA: 0s - loss: 0.6895 - accuracy: 0.7764 19/19 [==============================] - 0s 5ms/step - loss: 0.6799 - accuracy: 0.7850 Epoch 4/5 1/19 [>.............................] - ETA: 0s - loss: 0.4521 - accuracy: 0.8750 13/19 [===================>..........] - ETA: 0s - loss: 0.5492 - accuracy: 0.8197 19/19 [==============================] - 0s 5ms/step - loss: 0.5962 - accuracy: 0.8133 Epoch 5/5 1/19 [>.............................] - ETA: 0s - loss: 0.4230 - accuracy: 0.8438 13/19 [===================>..........] - ETA: 0s - loss: 0.4875 - accuracy: 0.8510 19/19 [==============================] - 0s 4ms/step - loss: 0.5158 - accuracy: 0.8400 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step [8 8 0 9 6 0 7 3 7 9 3 8 6 3 7 8 1 4 0 7 9 8 5 5 2 1 3 3 1 9 7 5 9 9 7 8 2 7 2 7 2 6 7 1 1 7 5 4 8 3 5 9 0 7 3 0 0 9 1 9] [8 8 0 9 2 0 7 3 7 9 3 8 4 3 7 8 1 4 0 7 9 8 5 5 2 1 3 4 6 7 7 5 9 9 7 8 2 7 4 7 0 3 5 1 1 5 5 2 8 3 5 9 0 7 3 0 0 7 1 9] .. GENERATED FROM PYTHON SOURCE LINES 212-214 We can see that even with just 5 epochs, the model already performs quite well. To quantify we can calculate the accuracy for this datapoint: .. GENERATED FROM PYTHON SOURCE LINES 214-218 .. code-block:: default from sklearn.metrics import accuracy_score accuracy_score(p1.predicted_labels_, FashionMNIST()[11].labels_as_array()) .. rst-class:: sphx-glr-script-out .. code-block:: none 0.8 .. GENERATED FROM PYTHON SOURCE LINES 219-225 Cross Validation ---------------- If we want to run a cross validation, we need to formalize the scoring into a function. We will calculate two types of accuracy: First, the accuracy per group and second, the accuracy over all images across all groups. For more information about how this works, check the :ref:`Custom Scorer ` example. .. GENERATED FROM PYTHON SOURCE LINES 225-248 .. code-block:: default from collections.abc import Sequence from tpcp.validate import Aggregator class SingleValueAccuracy(Aggregator[np.ndarray]): RETURN_RAW_SCORES = False @classmethod def aggregate(cls, /, values: Sequence[tuple[np.ndarray, np.ndarray]], **_) -> dict[str, float]: return {"accuracy": accuracy_score(np.hstack([v[0] for v in values]), np.hstack([v[1] for v in values]))} def scoring(pipeline, datapoint): result: np.ndarray = pipeline.safe_run(datapoint).predicted_labels_ reference = datapoint.labels_as_array() return { "accuracy": accuracy_score(result, reference), "per_sample": SingleValueAccuracy((result, reference)), } .. GENERATED FROM PYTHON SOURCE LINES 249-256 Now we can run a cross validation. We will only run it on a subset of the data, to keep the runtime manageable. .. note:: You might see warnings about retracing of the model. This is because we clone the pipeline before each call to the run method. This is a good idea to ensure that all pipelines are independent of each other, however, might result in some performance overhead. .. GENERATED FROM PYTHON SOURCE LINES 256-262 .. code-block:: default from tpcp.optimize import Optimize from tpcp.validate import cross_validate pipeline = KerasPipeline(n_train_epochs=10) cv_results = cross_validate(Optimize(pipeline), FashionMNIST()[:100], scoring=scoring, cv=3) .. rst-class:: sphx-glr-script-out .. code-block:: none CV Folds: 0%| | 0/3 [00:00............................] - ETA: 0s - loss: 1.8208 - accuracy: 0.3125  24/124 [====>.........................] - ETA: 0s - loss: 1.4874 - accuracy: 0.4570 36/124 [=======>......................] - ETA: 0s - loss: 1.2993 - accuracy: 0.5278 48/124 [==========>...................] - ETA: 0s - loss: 1.1716 - accuracy: 0.5794 60/124 [=============>................] - ETA: 0s - loss: 1.0816 - accuracy: 0.6141 72/124 [================>.............] - ETA: 0s - loss: 1.0366 - accuracy: 0.6319 84/124 [===================>..........] - ETA: 0s - loss: 0.9932 - accuracy: 0.6484 96/124 [======================>.......] - ETA: 0s - loss: 0.9648 - accuracy: 0.6621 108/124 [=========================>....] - ETA: 0s - loss: 0.9299 - accuracy: 0.6756 120/124 [============================>.] - ETA: 0s - loss: 0.9033 - accuracy: 0.6883 124/124 [==============================] - 1s 5ms/step - loss: 0.8968 - accuracy: 0.6919 Epoch 2/10 1/124 [..............................] - ETA: 0s - loss: 1.0481 - accuracy: 0.7500 13/124 [==>...........................] - ETA: 0s - loss: 0.6082 - accuracy: 0.8005 24/124 [====>.........................] - ETA: 0s - loss: 0.5817 - accuracy: 0.8112 36/124 [=======>......................] - ETA: 0s - loss: 0.6109 - accuracy: 0.7925 48/124 [==========>...................] - ETA: 0s - loss: 0.5921 - accuracy: 0.8008 61/124 [=============>................] - ETA: 0s - loss: 0.5840 - accuracy: 0.7976 73/124 [================>.............] - ETA: 0s - loss: 0.5668 - accuracy: 0.8074 85/124 [===================>..........] - ETA: 0s - loss: 0.5685 - accuracy: 0.8062 97/124 [======================>.......] - ETA: 0s - loss: 0.5629 - accuracy: 0.8115 109/124 [=========================>....] - ETA: 0s - loss: 0.5591 - accuracy: 0.8114 121/124 [============================>.] - ETA: 0s - loss: 0.5576 - accuracy: 0.8122 124/124 [==============================] - 1s 4ms/step - loss: 0.5606 - accuracy: 0.8104 Epoch 3/10 1/124 [..............................] - ETA: 0s - loss: 0.7180 - accuracy: 0.6250 13/124 [==>...........................] - ETA: 0s - loss: 0.5132 - accuracy: 0.8053 25/124 [=====>........................] - ETA: 0s - loss: 0.5200 - accuracy: 0.8125 37/124 [=======>......................] - ETA: 0s - loss: 0.5243 - accuracy: 0.8218 49/124 [==========>...................] - ETA: 0s - loss: 0.5135 - accuracy: 0.8233 61/124 [=============>................] - ETA: 0s - loss: 0.5053 - accuracy: 0.8222 73/124 [================>.............] - ETA: 0s - loss: 0.5060 - accuracy: 0.8215 85/124 [===================>..........] - ETA: 0s - loss: 0.4954 - accuracy: 0.8257 97/124 [======================>.......] - ETA: 0s - loss: 0.4940 - accuracy: 0.8264 109/124 [=========================>....] - ETA: 0s - loss: 0.4974 - accuracy: 0.8243 122/124 [============================>.] - ETA: 0s - loss: 0.4976 - accuracy: 0.8240 124/124 [==============================] - 1s 4ms/step - loss: 0.5009 - accuracy: 0.8232 Epoch 4/10 1/124 [..............................] - ETA: 0s - loss: 0.4630 - accuracy: 0.8125 13/124 [==>...........................] - ETA: 0s - loss: 0.3962 - accuracy: 0.8630 25/124 [=====>........................] - ETA: 0s - loss: 0.4104 - accuracy: 0.8587 37/124 [=======>......................] - ETA: 0s - loss: 0.4127 - accuracy: 0.8556 48/124 [==========>...................] - ETA: 0s - loss: 0.4300 - accuracy: 0.8522 58/124 [=============>................] - ETA: 0s - loss: 0.4337 - accuracy: 0.8491 69/124 [===============>..............] - ETA: 0s - loss: 0.4418 - accuracy: 0.8478 81/124 [==================>...........] - ETA: 0s - loss: 0.4444 - accuracy: 0.8484 93/124 [=====================>........] - ETA: 0s - loss: 0.4434 - accuracy: 0.8464 105/124 [========================>.....] - ETA: 0s - loss: 0.4405 - accuracy: 0.8467 117/124 [===========================>..] - ETA: 0s - loss: 0.4450 - accuracy: 0.8438 124/124 [==============================] - 1s 5ms/step - loss: 0.4448 - accuracy: 0.8437 Epoch 5/10 1/124 [..............................] - ETA: 0s - loss: 0.6924 - accuracy: 0.6250 13/124 [==>...........................] - ETA: 0s - loss: 0.4504 - accuracy: 0.8173 25/124 [=====>........................] - ETA: 0s - loss: 0.4423 - accuracy: 0.8263 37/124 [=======>......................] - ETA: 0s - loss: 0.4455 - accuracy: 0.8294 49/124 [==========>...................] - ETA: 0s - loss: 0.4360 - accuracy: 0.8399 61/124 [=============>................] - ETA: 0s - loss: 0.4225 - accuracy: 0.8438 73/124 [================>.............] - ETA: 0s - loss: 0.4221 - accuracy: 0.8485 85/124 [===================>..........] - ETA: 0s - loss: 0.4100 - accuracy: 0.8540 97/124 [======================>.......] - ETA: 0s - loss: 0.4067 - accuracy: 0.8560 109/124 [=========================>....] - ETA: 0s - loss: 0.4122 - accuracy: 0.8544 121/124 [============================>.] - ETA: 0s - loss: 0.4119 - accuracy: 0.8546 124/124 [==============================] - 1s 4ms/step - loss: 0.4119 - accuracy: 0.8540 Epoch 6/10 1/124 [..............................] - ETA: 0s - loss: 0.5976 - accuracy: 0.7812 14/124 [==>...........................] - ETA: 0s - loss: 0.3864 - accuracy: 0.8705 26/124 [=====>........................] - ETA: 0s - loss: 0.4092 - accuracy: 0.8558 38/124 [========>.....................] - ETA: 0s - loss: 0.3782 - accuracy: 0.8660 50/124 [===========>..................] - ETA: 0s - loss: 0.3883 - accuracy: 0.8650 62/124 [==============>...............] - ETA: 0s - loss: 0.3825 - accuracy: 0.8669 74/124 [================>.............] - ETA: 0s - loss: 0.3824 - accuracy: 0.8657 86/124 [===================>..........] - ETA: 0s - loss: 0.3877 - accuracy: 0.8645 98/124 [======================>.......] - ETA: 0s - loss: 0.3790 - accuracy: 0.8686 110/124 [=========================>....] - ETA: 0s - loss: 0.3823 - accuracy: 0.8673 122/124 [============================>.] - ETA: 0s - loss: 0.3810 - accuracy: 0.8678 124/124 [==============================] - 1s 4ms/step - loss: 0.3792 - accuracy: 0.8687 Epoch 7/10 1/124 [..............................] - ETA: 0s - loss: 0.4009 - accuracy: 0.8438 13/124 [==>...........................] - ETA: 0s - loss: 0.3694 - accuracy: 0.8726 25/124 [=====>........................] - ETA: 0s - loss: 0.3564 - accuracy: 0.8700 37/124 [=======>......................] - ETA: 0s - loss: 0.3387 - accuracy: 0.8758 49/124 [==========>...................] - ETA: 0s - loss: 0.3485 - accuracy: 0.8744 61/124 [=============>................] - ETA: 0s - loss: 0.3571 - accuracy: 0.8760 73/124 [================>.............] - ETA: 0s - loss: 0.3593 - accuracy: 0.8750 85/124 [===================>..........] - ETA: 0s - loss: 0.3578 - accuracy: 0.8757 97/124 [======================>.......] - ETA: 0s - loss: 0.3618 - accuracy: 0.8766 109/124 [=========================>....] - ETA: 0s - loss: 0.3580 - accuracy: 0.8767 121/124 [============================>.] - ETA: 0s - loss: 0.3582 - accuracy: 0.8755 124/124 [==============================] - 1s 4ms/step - loss: 0.3555 - accuracy: 0.8768 Epoch 8/10 1/124 [..............................] - ETA: 0s - loss: 0.2810 - accuracy: 0.9062 13/124 [==>...........................] - ETA: 0s - loss: 0.3236 - accuracy: 0.8966 25/124 [=====>........................] - ETA: 0s - loss: 0.2929 - accuracy: 0.9150 37/124 [=======>......................] - ETA: 0s - loss: 0.2969 - accuracy: 0.9079 49/124 [==========>...................] - ETA: 0s - loss: 0.3031 - accuracy: 0.9024 61/124 [=============>................] - ETA: 0s - loss: 0.3088 - accuracy: 0.8981 73/124 [================>.............] - ETA: 0s - loss: 0.3173 - accuracy: 0.8943 85/124 [===================>..........] - ETA: 0s - loss: 0.3251 - accuracy: 0.8926 97/124 [======================>.......] - ETA: 0s - loss: 0.3274 - accuracy: 0.8914 109/124 [=========================>....] - ETA: 0s - loss: 0.3369 - accuracy: 0.8879 121/124 [============================>.] - ETA: 0s - loss: 0.3466 - accuracy: 0.8825 124/124 [==============================] - 1s 4ms/step - loss: 0.3464 - accuracy: 0.8826 Epoch 9/10 1/124 [..............................] - ETA: 0s - loss: 0.2351 - accuracy: 0.9375 13/124 [==>...........................] - ETA: 0s - loss: 0.3713 - accuracy: 0.8726 25/124 [=====>........................] - ETA: 0s - loss: 0.3446 - accuracy: 0.8850 37/124 [=======>......................] - ETA: 0s - loss: 0.3441 - accuracy: 0.8742 49/124 [==========>...................] - ETA: 0s - loss: 0.3385 - accuracy: 0.8801 60/124 [=============>................] - ETA: 0s - loss: 0.3325 - accuracy: 0.8828 70/124 [===============>..............] - ETA: 0s - loss: 0.3258 - accuracy: 0.8862 82/124 [==================>...........] - ETA: 0s - loss: 0.3134 - accuracy: 0.8921 94/124 [=====================>........] - ETA: 0s - loss: 0.3241 - accuracy: 0.8893 106/124 [========================>.....] - ETA: 0s - loss: 0.3215 - accuracy: 0.8912 118/124 [===========================>..] - ETA: 0s - loss: 0.3186 - accuracy: 0.8893 124/124 [==============================] - 1s 5ms/step - loss: 0.3196 - accuracy: 0.8886 Epoch 10/10 1/124 [..............................] - ETA: 0s - loss: 0.2527 - accuracy: 0.9688 13/124 [==>...........................] - ETA: 0s - loss: 0.2215 - accuracy: 0.9255 25/124 [=====>........................] - ETA: 0s - loss: 0.2457 - accuracy: 0.9125 37/124 [=======>......................] - ETA: 0s - loss: 0.2692 - accuracy: 0.9037 49/124 [==========>...................] - ETA: 0s - loss: 0.2755 - accuracy: 0.9031 61/124 [=============>................] - ETA: 0s - loss: 0.2910 - accuracy: 0.9001 73/124 [================>.............] - ETA: 0s - loss: 0.2991 - accuracy: 0.8951 85/124 [===================>..........] - ETA: 0s - loss: 0.2970 - accuracy: 0.8960 97/124 [======================>.......] - ETA: 0s - loss: 0.2953 - accuracy: 0.8982 109/124 [=========================>....] - ETA: 0s - loss: 0.2941 - accuracy: 0.8999 121/124 [============================>.] - ETA: 0s - loss: 0.2929 - accuracy: 0.8993 124/124 [==============================] - 1s 4ms/step - loss: 0.2921 - accuracy: 0.8997 Datapoints: 0%| | 0/34 [00:00...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 3%|▎ | 1/34 [00:00<00:09, 3.54it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 6%|▌ | 2/34 [00:00<00:09, 3.46it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 9%|▉ | 3/34 [00:00<00:08, 3.47it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 12%|█▏ | 4/34 [00:01<00:08, 3.44it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 15%|█▍ | 5/34 [00:01<00:08, 3.48it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 18%|█▊ | 6/34 [00:01<00:08, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 21%|██ | 7/34 [00:02<00:07, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 24%|██▎ | 8/34 [00:02<00:07, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 26%|██▋ | 9/34 [00:02<00:07, 3.39it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 29%|██▉ | 10/34 [00:02<00:07, 3.39it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 32%|███▏ | 11/34 [00:03<00:06, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 35%|███▌ | 12/34 [00:03<00:06, 3.38it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 38%|███▊ | 13/34 [00:03<00:06, 3.39it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 41%|████ | 14/34 [00:04<00:05, 3.40it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 44%|████▍ | 15/34 [00:04<00:05, 3.38it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 47%|████▋ | 16/34 [00:04<00:05, 3.43it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 50%|█████ | 17/34 [00:04<00:04, 3.47it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 53%|█████▎ | 18/34 [00:05<00:04, 3.43it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 56%|█████▌ | 19/34 [00:05<00:04, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 59%|█████▉ | 20/34 [00:05<00:04, 3.41it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 62%|██████▏ | 21/34 [00:06<00:03, 3.39it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 65%|██████▍ | 22/34 [00:06<00:03, 3.39it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 68%|██████▊ | 23/34 [00:06<00:03, 3.40it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 71%|███████ | 24/34 [00:07<00:02, 3.37it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 74%|███████▎ | 25/34 [00:07<00:02, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 76%|███████▋ | 26/34 [00:07<00:02, 3.47it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 79%|███████▉ | 27/34 [00:07<00:02, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 82%|████████▏ | 28/34 [00:08<00:01, 3.41it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 85%|████████▌ | 29/34 [00:08<00:01, 3.41it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 88%|████████▊ | 30/34 [00:08<00:01, 3.37it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 91%|█████████ | 31/34 [00:09<00:00, 3.37it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 94%|█████████▍| 32/34 [00:09<00:00, 3.43it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 97%|█████████▋| 33/34 [00:09<00:00, 3.40it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 100%|██████████| 34/34 [00:09<00:00, 3.39it/s] Datapoints: 100%|██████████| 34/34 [00:09<00:00, 3.41it/s] CV Folds: 33%|███▎ | 1/3 [00:22<00:45, 22.57s/it](4020, 28, 28) Epoch 1/10 1/126 [..............................] - ETA: 1:55 - loss: 2.3656 - accuracy: 0.1250 13/126 [==>...........................] - ETA: 0s - loss: 1.7826 - accuracy: 0.3870  25/126 [====>.........................] - ETA: 0s - loss: 1.4578 - accuracy: 0.5150 37/126 [=======>......................] - ETA: 0s - loss: 1.3063 - accuracy: 0.5633 49/126 [==========>...................] - ETA: 0s - loss: 1.2005 - accuracy: 0.6014 61/126 [=============>................] - ETA: 0s - loss: 1.1294 - accuracy: 0.6168 73/126 [================>.............] - ETA: 0s - loss: 1.0706 - accuracy: 0.6378 85/126 [===================>..........] - ETA: 0s - loss: 1.0088 - accuracy: 0.6603 97/126 [======================>.......] - ETA: 0s - loss: 0.9718 - accuracy: 0.6727 109/126 [========================>.....] - ETA: 0s - loss: 0.9331 - accuracy: 0.6841 121/126 [===========================>..] - ETA: 0s - loss: 0.9028 - accuracy: 0.6955 126/126 [==============================] - 1s 4ms/step - loss: 0.8913 - accuracy: 0.7000 Epoch 2/10 1/126 [..............................] - ETA: 0s - loss: 0.4984 - accuracy: 0.8125 13/126 [==>...........................] - ETA: 0s - loss: 0.5529 - accuracy: 0.7957 25/126 [====>.........................] - ETA: 0s - loss: 0.5682 - accuracy: 0.8037 37/126 [=======>......................] - ETA: 0s - loss: 0.5658 - accuracy: 0.7990 49/126 [==========>...................] - ETA: 0s - loss: 0.5778 - accuracy: 0.7927 61/126 [=============>................] - ETA: 0s - loss: 0.5962 - accuracy: 0.7874 73/126 [================>.............] - ETA: 0s - loss: 0.5902 - accuracy: 0.7902 85/126 [===================>..........] - ETA: 0s - loss: 0.5773 - accuracy: 0.7926 97/126 [======================>.......] - ETA: 0s - loss: 0.5772 - accuracy: 0.7938 109/126 [========================>.....] - ETA: 0s - loss: 0.5713 - accuracy: 0.7967 121/126 [===========================>..] - ETA: 0s - loss: 0.5649 - accuracy: 0.7998 126/126 [==============================] - 1s 4ms/step - loss: 0.5662 - accuracy: 0.8002 Epoch 3/10 1/126 [..............................] - ETA: 0s - loss: 0.5249 - accuracy: 0.8125 13/126 [==>...........................] - ETA: 0s - loss: 0.5380 - accuracy: 0.8125 25/126 [====>.........................] - ETA: 0s - loss: 0.5201 - accuracy: 0.8275 37/126 [=======>......................] - ETA: 0s - loss: 0.4840 - accuracy: 0.8353 49/126 [==========>...................] - ETA: 0s - loss: 0.4717 - accuracy: 0.8329 61/126 [=============>................] - ETA: 0s - loss: 0.4678 - accuracy: 0.8381 73/126 [================>.............] - ETA: 0s - loss: 0.4692 - accuracy: 0.8373 85/126 [===================>..........] - ETA: 0s - loss: 0.4810 - accuracy: 0.8342 97/126 [======================>.......] - ETA: 0s - loss: 0.4782 - accuracy: 0.8354 109/126 [========================>.....] - ETA: 0s - loss: 0.4805 - accuracy: 0.8343 121/126 [===========================>..] - ETA: 0s - loss: 0.4901 - accuracy: 0.8275 126/126 [==============================] - 1s 4ms/step - loss: 0.4893 - accuracy: 0.8279 Epoch 4/10 1/126 [..............................] - ETA: 0s - loss: 0.2082 - accuracy: 0.9688 13/126 [==>...........................] - ETA: 0s - loss: 0.4042 - accuracy: 0.8654 25/126 [====>.........................] - ETA: 0s - loss: 0.4238 - accuracy: 0.8587 37/126 [=======>......................] - ETA: 0s - loss: 0.4437 - accuracy: 0.8480 49/126 [==========>...................] - ETA: 0s - loss: 0.4393 - accuracy: 0.8469 61/126 [=============>................] - ETA: 0s - loss: 0.4340 - accuracy: 0.8478 73/126 [================>.............] - ETA: 0s - loss: 0.4283 - accuracy: 0.8527 85/126 [===================>..........] - ETA: 0s - loss: 0.4348 - accuracy: 0.8526 97/126 [======================>.......] - ETA: 0s - loss: 0.4394 - accuracy: 0.8502 109/126 [========================>.....] - ETA: 0s - loss: 0.4472 - accuracy: 0.8438 121/126 [===========================>..] - ETA: 0s - loss: 0.4457 - accuracy: 0.8450 126/126 [==============================] - 1s 4ms/step - loss: 0.4431 - accuracy: 0.8453 Epoch 5/10 1/126 [..............................] - ETA: 0s - loss: 0.1979 - accuracy: 0.9375 13/126 [==>...........................] - ETA: 0s - loss: 0.3542 - accuracy: 0.8822 25/126 [====>.........................] - ETA: 0s - loss: 0.3946 - accuracy: 0.8612 37/126 [=======>......................] - ETA: 0s - loss: 0.4174 - accuracy: 0.8471 49/126 [==========>...................] - ETA: 0s - loss: 0.4057 - accuracy: 0.8540 61/126 [=============>................] - ETA: 0s - loss: 0.4065 - accuracy: 0.8535 73/126 [================>.............] - ETA: 0s - loss: 0.4017 - accuracy: 0.8570 85/126 [===================>..........] - ETA: 0s - loss: 0.4044 - accuracy: 0.8570 97/126 [======================>.......] - ETA: 0s - loss: 0.4087 - accuracy: 0.8544 109/126 [========================>.....] - ETA: 0s - loss: 0.4048 - accuracy: 0.8569 121/126 [===========================>..] - ETA: 0s - loss: 0.4056 - accuracy: 0.8574 126/126 [==============================] - 1s 4ms/step - loss: 0.4046 - accuracy: 0.8580 Epoch 6/10 1/126 [..............................] - ETA: 0s - loss: 0.4598 - accuracy: 0.8125 13/126 [==>...........................] - ETA: 0s - loss: 0.3434 - accuracy: 0.8534 25/126 [====>.........................] - ETA: 0s - loss: 0.3639 - accuracy: 0.8625 37/126 [=======>......................] - ETA: 0s - loss: 0.3791 - accuracy: 0.8573 49/126 [==========>...................] - ETA: 0s - loss: 0.3696 - accuracy: 0.8642 61/126 [=============>................] - ETA: 0s - loss: 0.3709 - accuracy: 0.8622 73/126 [================>.............] - ETA: 0s - loss: 0.3741 - accuracy: 0.8647 85/126 [===================>..........] - ETA: 0s - loss: 0.3770 - accuracy: 0.8654 97/126 [======================>.......] - ETA: 0s - loss: 0.3796 - accuracy: 0.8650 109/126 [========================>.....] - ETA: 0s - loss: 0.3698 - accuracy: 0.8687 121/126 [===========================>..] - ETA: 0s - loss: 0.3775 - accuracy: 0.8662 126/126 [==============================] - 1s 4ms/step - loss: 0.3772 - accuracy: 0.8667 Epoch 7/10 1/126 [..............................] - ETA: 0s - loss: 0.2244 - accuracy: 0.9062 13/126 [==>...........................] - ETA: 0s - loss: 0.3541 - accuracy: 0.8798 25/126 [====>.........................] - ETA: 0s - loss: 0.3532 - accuracy: 0.8813 37/126 [=======>......................] - ETA: 0s - loss: 0.3349 - accuracy: 0.8843 49/126 [==========>...................] - ETA: 0s - loss: 0.3264 - accuracy: 0.8884 61/126 [=============>................] - ETA: 0s - loss: 0.3308 - accuracy: 0.8858 73/126 [================>.............] - ETA: 0s - loss: 0.3448 - accuracy: 0.8823 85/126 [===================>..........] - ETA: 0s - loss: 0.3522 - accuracy: 0.8790 97/126 [======================>.......] - ETA: 0s - loss: 0.3469 - accuracy: 0.8798 109/126 [========================>.....] - ETA: 0s - loss: 0.3480 - accuracy: 0.8802 121/126 [===========================>..] - ETA: 0s - loss: 0.3487 - accuracy: 0.8786 126/126 [==============================] - 1s 4ms/step - loss: 0.3479 - accuracy: 0.8786 Epoch 8/10 1/126 [..............................] - ETA: 0s - loss: 0.2092 - accuracy: 0.9375 13/126 [==>...........................] - ETA: 0s - loss: 0.3092 - accuracy: 0.8942 25/126 [====>.........................] - ETA: 0s - loss: 0.2997 - accuracy: 0.9000 37/126 [=======>......................] - ETA: 0s - loss: 0.2936 - accuracy: 0.8986 49/126 [==========>...................] - ETA: 0s - loss: 0.3115 - accuracy: 0.8935 61/126 [=============>................] - ETA: 0s - loss: 0.3090 - accuracy: 0.8934 73/126 [================>.............] - ETA: 0s - loss: 0.3184 - accuracy: 0.8887 85/126 [===================>..........] - ETA: 0s - loss: 0.3252 - accuracy: 0.8875 97/126 [======================>.......] - ETA: 0s - loss: 0.3264 - accuracy: 0.8869 109/126 [========================>.....] - ETA: 0s - loss: 0.3268 - accuracy: 0.8885 121/126 [===========================>..] - ETA: 0s - loss: 0.3287 - accuracy: 0.8861 126/126 [==============================] - 1s 4ms/step - loss: 0.3285 - accuracy: 0.8863 Epoch 9/10 1/126 [..............................] - ETA: 0s - loss: 0.1886 - accuracy: 0.9062 13/126 [==>...........................] - ETA: 0s - loss: 0.2965 - accuracy: 0.9038 25/126 [====>.........................] - ETA: 0s - loss: 0.2998 - accuracy: 0.9000 37/126 [=======>......................] - ETA: 0s - loss: 0.2976 - accuracy: 0.8953 49/126 [==========>...................] - ETA: 0s - loss: 0.3057 - accuracy: 0.8909 61/126 [=============>................] - ETA: 0s - loss: 0.3055 - accuracy: 0.8842 73/126 [================>.............] - ETA: 0s - loss: 0.3189 - accuracy: 0.8789 85/126 [===================>..........] - ETA: 0s - loss: 0.3181 - accuracy: 0.8809 97/126 [======================>.......] - ETA: 0s - loss: 0.3062 - accuracy: 0.8860 109/126 [========================>.....] - ETA: 0s - loss: 0.3139 - accuracy: 0.8862 121/126 [===========================>..] - ETA: 0s - loss: 0.3122 - accuracy: 0.8892 126/126 [==============================] - 1s 4ms/step - loss: 0.3120 - accuracy: 0.8900 Epoch 10/10 1/126 [..............................] - ETA: 0s - loss: 0.2327 - accuracy: 0.9062 13/126 [==>...........................] - ETA: 0s - loss: 0.2908 - accuracy: 0.8870 25/126 [====>.........................] - ETA: 0s - loss: 0.2960 - accuracy: 0.8825 37/126 [=======>......................] - ETA: 0s - loss: 0.3081 - accuracy: 0.8851 49/126 [==========>...................] - ETA: 0s - loss: 0.2985 - accuracy: 0.8903 61/126 [=============>................] - ETA: 0s - loss: 0.2978 - accuracy: 0.8888 73/126 [================>.............] - ETA: 0s - loss: 0.3040 - accuracy: 0.8883 85/126 [===================>..........] - ETA: 0s - loss: 0.3004 - accuracy: 0.8886 97/126 [======================>.......] - ETA: 0s - loss: 0.3038 - accuracy: 0.8876 109/126 [========================>.....] - ETA: 0s - loss: 0.3052 - accuracy: 0.8882 121/126 [===========================>..] - ETA: 0s - loss: 0.3039 - accuracy: 0.8882 126/126 [==============================] - 1s 4ms/step - loss: 0.3065 - accuracy: 0.8871 Datapoints: 0%| | 0/33 [00:00...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 3%|▎ | 1/33 [00:00<00:08, 3.58it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 2ms/step Datapoints: 6%|▌ | 2/33 [00:00<00:08, 3.48it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 9%|▉ | 3/33 [00:00<00:08, 3.45it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 12%|█▏ | 4/33 [00:01<00:08, 3.40it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 15%|█▌ | 5/33 [00:01<00:08, 3.44it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 18%|█▊ | 6/33 [00:01<00:07, 3.43it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 21%|██ | 7/33 [00:02<00:07, 3.45it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 24%|██▍ | 8/33 [00:02<00:07, 3.44it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 27%|██▋ | 9/33 [00:02<00:07, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 30%|███ | 10/33 [00:02<00:06, 3.38it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 33%|███▎ | 11/33 [00:03<00:06, 3.38it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 36%|███▋ | 12/33 [00:03<00:06, 3.39it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 39%|███▉ | 13/33 [00:03<00:05, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 2ms/step Datapoints: 42%|████▏ | 14/33 [00:04<00:05, 3.43it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 45%|████▌ | 15/33 [00:04<00:05, 3.41it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 48%|████▊ | 16/33 [00:04<00:05, 3.38it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 52%|█████▏ | 17/33 [00:04<00:04, 3.39it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 55%|█████▍ | 18/33 [00:05<00:04, 3.44it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 58%|█████▊ | 19/33 [00:05<00:04, 3.45it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 2ms/step Datapoints: 61%|██████ | 20/33 [00:05<00:03, 3.48it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 64%|██████▎ | 21/33 [00:06<00:03, 3.52it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 67%|██████▋ | 22/33 [00:06<00:03, 3.45it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 70%|██████▉ | 23/33 [00:06<00:02, 3.43it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 73%|███████▎ | 24/33 [00:06<00:02, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 2ms/step Datapoints: 76%|███████▌ | 25/33 [00:07<00:02, 3.44it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 79%|███████▉ | 26/33 [00:07<00:02, 3.48it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 82%|████████▏ | 27/33 [00:07<00:01, 3.51it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 85%|████████▍ | 28/33 [00:08<00:01, 3.45it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 88%|████████▊ | 29/33 [00:08<00:01, 3.43it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 91%|█████████ | 30/33 [00:08<00:00, 3.48it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 94%|█████████▍| 31/33 [00:09<00:00, 3.48it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 97%|█████████▋| 32/33 [00:09<00:00, 3.51it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 100%|██████████| 33/33 [00:09<00:00, 3.52it/s] Datapoints: 100%|██████████| 33/33 [00:09<00:00, 3.45it/s] CV Folds: 67%|██████▋ | 2/3 [00:44<00:22, 22.31s/it](4020, 28, 28) Epoch 1/10 1/126 [..............................] - ETA: 1:56 - loss: 2.3129 - accuracy: 0.0938 12/126 [=>............................] - ETA: 0s - loss: 1.7505 - accuracy: 0.3984  24/126 [====>.........................] - ETA: 0s - loss: 1.4718 - accuracy: 0.5026 36/126 [=======>......................] - ETA: 0s - loss: 1.3248 - accuracy: 0.5460 48/126 [==========>...................] - ETA: 0s - loss: 1.2118 - accuracy: 0.5905 60/126 [=============>................] - ETA: 0s - loss: 1.1265 - accuracy: 0.6182 72/126 [================>.............] - ETA: 0s - loss: 1.0826 - accuracy: 0.6328 84/126 [===================>..........] - ETA: 0s - loss: 1.0169 - accuracy: 0.6562 96/126 [=====================>........] - ETA: 0s - loss: 0.9770 - accuracy: 0.6712 108/126 [========================>.....] - ETA: 0s - loss: 0.9428 - accuracy: 0.6808 120/126 [===========================>..] - ETA: 0s - loss: 0.9144 - accuracy: 0.6922 126/126 [==============================] - 1s 4ms/step - loss: 0.8965 - accuracy: 0.6983 Epoch 2/10 1/126 [..............................] - ETA: 0s - loss: 0.4769 - accuracy: 0.8125 13/126 [==>...........................] - ETA: 0s - loss: 0.5761 - accuracy: 0.7957 25/126 [====>.........................] - ETA: 0s - loss: 0.6120 - accuracy: 0.7912 37/126 [=======>......................] - ETA: 0s - loss: 0.6063 - accuracy: 0.7889 49/126 [==========>...................] - ETA: 0s - loss: 0.5931 - accuracy: 0.7972 61/126 [=============>................] - ETA: 0s - loss: 0.5958 - accuracy: 0.7935 73/126 [================>.............] - ETA: 0s - loss: 0.5951 - accuracy: 0.7954 85/126 [===================>..........] - ETA: 0s - loss: 0.5894 - accuracy: 0.7967 97/126 [======================>.......] - ETA: 0s - loss: 0.5817 - accuracy: 0.8015 109/126 [========================>.....] - ETA: 0s - loss: 0.5710 - accuracy: 0.8048 121/126 [===========================>..] - ETA: 0s - loss: 0.5654 - accuracy: 0.8068 126/126 [==============================] - 1s 4ms/step - loss: 0.5750 - accuracy: 0.8040 Epoch 3/10 1/126 [..............................] - ETA: 0s - loss: 0.6893 - accuracy: 0.7812 13/126 [==>...........................] - ETA: 0s - loss: 0.5276 - accuracy: 0.8245 25/126 [====>.........................] - ETA: 0s - loss: 0.5681 - accuracy: 0.8075 37/126 [=======>......................] - ETA: 0s - loss: 0.5517 - accuracy: 0.8083 49/126 [==========>...................] - ETA: 0s - loss: 0.5377 - accuracy: 0.8119 61/126 [=============>................] - ETA: 0s - loss: 0.5310 - accuracy: 0.8166 73/126 [================>.............] - ETA: 0s - loss: 0.5269 - accuracy: 0.8185 85/126 [===================>..........] - ETA: 0s - loss: 0.5284 - accuracy: 0.8210 97/126 [======================>.......] - ETA: 0s - loss: 0.5211 - accuracy: 0.8212 109/126 [========================>.....] - ETA: 0s - loss: 0.5114 - accuracy: 0.8248 121/126 [===========================>..] - ETA: 0s - loss: 0.5134 - accuracy: 0.8236 126/126 [==============================] - 1s 4ms/step - loss: 0.5140 - accuracy: 0.8224 Epoch 4/10 1/126 [..............................] - ETA: 0s - loss: 0.4519 - accuracy: 0.9062 13/126 [==>...........................] - ETA: 0s - loss: 0.4019 - accuracy: 0.8750 25/126 [====>.........................] - ETA: 0s - loss: 0.4193 - accuracy: 0.8587 37/126 [=======>......................] - ETA: 0s - loss: 0.4405 - accuracy: 0.8480 49/126 [==========>...................] - ETA: 0s - loss: 0.4532 - accuracy: 0.8399 61/126 [=============>................] - ETA: 0s - loss: 0.4511 - accuracy: 0.8412 73/126 [================>.............] - ETA: 0s - loss: 0.4450 - accuracy: 0.8438 85/126 [===================>..........] - ETA: 0s - loss: 0.4464 - accuracy: 0.8438 97/126 [======================>.......] - ETA: 0s - loss: 0.4548 - accuracy: 0.8415 109/126 [========================>.....] - ETA: 0s - loss: 0.4619 - accuracy: 0.8394 121/126 [===========================>..] - ETA: 0s - loss: 0.4629 - accuracy: 0.8383 126/126 [==============================] - 1s 4ms/step - loss: 0.4667 - accuracy: 0.8358 Epoch 5/10 1/126 [..............................] - ETA: 0s - loss: 0.4192 - accuracy: 0.8125 13/126 [==>...........................] - ETA: 0s - loss: 0.4263 - accuracy: 0.8654 25/126 [====>.........................] - ETA: 0s - loss: 0.4182 - accuracy: 0.8625 37/126 [=======>......................] - ETA: 0s - loss: 0.4131 - accuracy: 0.8556 49/126 [==========>...................] - ETA: 0s - loss: 0.4216 - accuracy: 0.8635 61/126 [=============>................] - ETA: 0s - loss: 0.4160 - accuracy: 0.8648 73/126 [================>.............] - ETA: 0s - loss: 0.4160 - accuracy: 0.8634 85/126 [===================>..........] - ETA: 0s - loss: 0.4222 - accuracy: 0.8599 97/126 [======================>.......] - ETA: 0s - loss: 0.4279 - accuracy: 0.8595 109/126 [========================>.....] - ETA: 0s - loss: 0.4202 - accuracy: 0.8595 121/126 [===========================>..] - ETA: 0s - loss: 0.4177 - accuracy: 0.8598 126/126 [==============================] - 1s 4ms/step - loss: 0.4173 - accuracy: 0.8600 Epoch 6/10 1/126 [..............................] - ETA: 0s - loss: 0.8799 - accuracy: 0.7500 13/126 [==>...........................] - ETA: 0s - loss: 0.3963 - accuracy: 0.8582 25/126 [====>.........................] - ETA: 0s - loss: 0.3716 - accuracy: 0.8662 37/126 [=======>......................] - ETA: 0s - loss: 0.3773 - accuracy: 0.8674 49/126 [==========>...................] - ETA: 0s - loss: 0.3820 - accuracy: 0.8699 60/126 [=============>................] - ETA: 0s - loss: 0.3773 - accuracy: 0.8672 72/126 [================>.............] - ETA: 0s - loss: 0.3877 - accuracy: 0.8672 84/126 [===================>..........] - ETA: 0s - loss: 0.3925 - accuracy: 0.8650 96/126 [=====================>........] - ETA: 0s - loss: 0.3944 - accuracy: 0.8620 108/126 [========================>.....] - ETA: 0s - loss: 0.3934 - accuracy: 0.8623 120/126 [===========================>..] - ETA: 0s - loss: 0.3980 - accuracy: 0.8628 126/126 [==============================] - 1s 4ms/step - loss: 0.3949 - accuracy: 0.8624 Epoch 7/10 1/126 [..............................] - ETA: 0s - loss: 0.2022 - accuracy: 0.9375 13/126 [==>...........................] - ETA: 0s - loss: 0.3838 - accuracy: 0.8774 25/126 [====>.........................] - ETA: 0s - loss: 0.3768 - accuracy: 0.8775 37/126 [=======>......................] - ETA: 0s - loss: 0.3513 - accuracy: 0.8801 49/126 [==========>...................] - ETA: 0s - loss: 0.3411 - accuracy: 0.8852 61/126 [=============>................] - ETA: 0s - loss: 0.3486 - accuracy: 0.8781 73/126 [================>.............] - ETA: 0s - loss: 0.3523 - accuracy: 0.8780 85/126 [===================>..........] - ETA: 0s - loss: 0.3631 - accuracy: 0.8757 97/126 [======================>.......] - ETA: 0s - loss: 0.3614 - accuracy: 0.8769 109/126 [========================>.....] - ETA: 0s - loss: 0.3655 - accuracy: 0.8761 121/126 [===========================>..] - ETA: 0s - loss: 0.3640 - accuracy: 0.8771 126/126 [==============================] - 1s 4ms/step - loss: 0.3635 - accuracy: 0.8774 Epoch 8/10 1/126 [..............................] - ETA: 0s - loss: 0.3186 - accuracy: 0.8750 13/126 [==>...........................] - ETA: 0s - loss: 0.3331 - accuracy: 0.8846 25/126 [====>.........................] - ETA: 0s - loss: 0.3351 - accuracy: 0.8850 37/126 [=======>......................] - ETA: 0s - loss: 0.3366 - accuracy: 0.8843 49/126 [==========>...................] - ETA: 0s - loss: 0.3372 - accuracy: 0.8852 61/126 [=============>................] - ETA: 0s - loss: 0.3334 - accuracy: 0.8842 73/126 [================>.............] - ETA: 0s - loss: 0.3354 - accuracy: 0.8848 85/126 [===================>..........] - ETA: 0s - loss: 0.3371 - accuracy: 0.8860 97/126 [======================>.......] - ETA: 0s - loss: 0.3355 - accuracy: 0.8860 109/126 [========================>.....] - ETA: 0s - loss: 0.3349 - accuracy: 0.8845 121/126 [===========================>..] - ETA: 0s - loss: 0.3427 - accuracy: 0.8830 126/126 [==============================] - 1s 4ms/step - loss: 0.3457 - accuracy: 0.8826 Epoch 9/10 1/126 [..............................] - ETA: 0s - loss: 0.2176 - accuracy: 0.9375 13/126 [==>...........................] - ETA: 0s - loss: 0.3388 - accuracy: 0.8822 25/126 [====>.........................] - ETA: 0s - loss: 0.3464 - accuracy: 0.8838 37/126 [=======>......................] - ETA: 0s - loss: 0.3463 - accuracy: 0.8742 49/126 [==========>...................] - ETA: 0s - loss: 0.3446 - accuracy: 0.8737 61/126 [=============>................] - ETA: 0s - loss: 0.3297 - accuracy: 0.8811 73/126 [================>.............] - ETA: 0s - loss: 0.3296 - accuracy: 0.8827 85/126 [===================>..........] - ETA: 0s - loss: 0.3323 - accuracy: 0.8816 97/126 [======================>.......] - ETA: 0s - loss: 0.3263 - accuracy: 0.8860 109/126 [========================>.....] - ETA: 0s - loss: 0.3284 - accuracy: 0.8873 121/126 [===========================>..] - ETA: 0s - loss: 0.3296 - accuracy: 0.8874 126/126 [==============================] - 1s 4ms/step - loss: 0.3285 - accuracy: 0.8878 Epoch 10/10 1/126 [..............................] - ETA: 0s - loss: 0.3079 - accuracy: 0.8125 13/126 [==>...........................] - ETA: 0s - loss: 0.2863 - accuracy: 0.9087 25/126 [====>.........................] - ETA: 0s - loss: 0.2799 - accuracy: 0.9075 37/126 [=======>......................] - ETA: 0s - loss: 0.2985 - accuracy: 0.9054 49/126 [==========>...................] - ETA: 0s - loss: 0.2956 - accuracy: 0.9037 67/126 [==============>...............] - ETA: 0s - loss: 0.2908 - accuracy: 0.9053 79/126 [=================>............] - ETA: 0s - loss: 0.2832 - accuracy: 0.9066 91/126 [====================>.........] - ETA: 0s - loss: 0.2922 - accuracy: 0.9052 103/126 [=======================>......] - ETA: 0s - loss: 0.2963 - accuracy: 0.9017 115/126 [==========================>...] - ETA: 0s - loss: 0.2983 - accuracy: 0.9000 126/126 [==============================] - 1s 4ms/step - loss: 0.3008 - accuracy: 0.8993 Datapoints: 0%| | 0/33 [00:00...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 3%|▎ | 1/33 [00:00<00:08, 3.58it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 6%|▌ | 2/33 [00:00<00:08, 3.46it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 9%|▉ | 3/33 [00:00<00:08, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 12%|█▏ | 4/33 [00:01<00:08, 3.35it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 15%|█▌ | 5/33 [00:01<00:08, 3.36it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 18%|█▊ | 6/33 [00:01<00:08, 3.37it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 21%|██ | 7/33 [00:02<00:07, 3.40it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 24%|██▍ | 8/33 [00:02<00:07, 3.46it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 2ms/step Datapoints: 27%|██▋ | 9/33 [00:02<00:06, 3.44it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 30%|███ | 10/33 [00:02<00:06, 3.41it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 33%|███▎ | 11/33 [00:03<00:06, 3.41it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 36%|███▋ | 12/33 [00:03<00:06, 3.40it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 39%|███▉ | 13/33 [00:03<00:05, 3.38it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 42%|████▏ | 14/33 [00:04<00:05, 3.37it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 45%|████▌ | 15/33 [00:04<00:05, 3.38it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 48%|████▊ | 16/33 [00:04<00:04, 3.40it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 52%|█████▏ | 17/33 [00:04<00:04, 3.40it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 2ms/step Datapoints: 55%|█████▍ | 18/33 [00:05<00:04, 3.40it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 58%|█████▊ | 19/33 [00:05<00:04, 3.43it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 61%|██████ | 20/33 [00:05<00:03, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 64%|██████▎ | 21/33 [00:06<00:03, 3.46it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 67%|██████▋ | 22/33 [00:06<00:03, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 70%|██████▉ | 23/33 [00:06<00:02, 3.41it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 73%|███████▎ | 24/33 [00:07<00:02, 3.44it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 76%|███████▌ | 25/33 [00:07<00:02, 3.44it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 79%|███████▉ | 26/33 [00:07<00:02, 3.42it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 82%|████████▏ | 27/33 [00:07<00:01, 3.46it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 85%|████████▍ | 28/33 [00:08<00:01, 3.39it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 88%|████████▊ | 29/33 [00:08<00:01, 3.39it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 91%|█████████ | 30/33 [00:08<00:00, 3.40it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 94%|█████████▍| 31/33 [00:09<00:00, 3.40it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 97%|█████████▋| 32/33 [00:09<00:00, 3.38it/s] 1/2 [==============>...............] - ETA: 0s 2/2 [==============================] - 0s 3ms/step Datapoints: 100%|██████████| 33/33 [00:09<00:00, 3.39it/s] Datapoints: 100%|██████████| 33/33 [00:09<00:00, 3.41it/s] CV Folds: 100%|██████████| 3/3 [01:02<00:00, 20.17s/it] CV Folds: 100%|██████████| 3/3 [01:02<00:00, 20.78s/it] .. GENERATED FROM PYTHON SOURCE LINES 263-264 We can now look at the results per group: .. GENERATED FROM PYTHON SOURCE LINES 264-266 .. code-block:: default cv_results["test_single_accuracy"] .. rst-class:: sphx-glr-script-out .. code-block:: none [[0.85, 0.8666666666666667, 0.9333333333333333, 0.8, 0.85, 0.85, 0.85, 0.8833333333333333, 0.85, 0.85, 0.8166666666666667, 0.8666666666666667, 0.9333333333333333, 0.8, 0.8833333333333333, 0.8, 0.8, 0.8833333333333333, 0.7833333333333333, 0.8166666666666667, 0.8, 0.85, 0.8, 0.95, 0.8, 0.8833333333333333, 0.8833333333333333, 0.85, 0.85, 0.7833333333333333, 0.75, 0.8666666666666667, 0.8333333333333334, 0.9333333333333333], [0.7833333333333333, 0.8166666666666667, 0.7666666666666667, 0.7833333333333333, 0.85, 0.8, 0.8, 0.8166666666666667, 0.8, 0.8333333333333334, 0.8333333333333334, 0.8, 0.8166666666666667, 0.85, 0.75, 0.7, 0.8333333333333334, 0.8166666666666667, 0.8666666666666667, 0.7666666666666667, 0.7833333333333333, 0.7833333333333333, 0.7666666666666667, 0.8833333333333333, 0.8333333333333334, 0.7833333333333333, 0.8, 0.85, 0.85, 0.8166666666666667, 0.7666666666666667, 0.8, 0.8833333333333333], [0.7666666666666667, 0.9166666666666666, 0.8, 0.8, 0.7833333333333333, 0.85, 0.8833333333333333, 0.85, 0.9, 0.8833333333333333, 0.9, 0.85, 0.8833333333333333, 0.8166666666666667, 0.8333333333333334, 0.8333333333333334, 0.85, 0.7833333333333333, 0.7166666666666667, 0.7333333333333333, 0.75, 0.7833333333333333, 0.85, 0.7666666666666667, 0.7833333333333333, 0.9, 0.8666666666666667, 0.8333333333333334, 0.8333333333333334, 0.8166666666666667, 0.85, 0.85, 0.9]] .. GENERATED FROM PYTHON SOURCE LINES 267-268 And the overall accuracy as the average over all samples of all groups within a fold: .. GENERATED FROM PYTHON SOURCE LINES 268-269 .. code-block:: default cv_results["test_per_sample__accuracy"] .. rst-class:: sphx-glr-script-out .. code-block:: none array([0.84705882, 0.80858586, 0.83080808]) .. rst-class:: sphx-glr-timing **Total running time of the script:** (1 minutes 9.074 seconds) **Estimated memory usage:** 278 MB .. _sphx_glr_download_auto_examples_integrations__01_tensorflow.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: _01_tensorflow.py <_01_tensorflow.py>` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: _01_tensorflow.ipynb <_01_tensorflow.ipynb>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_