CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutSign UpSign In
better-data-science

CoCalc provides the best real-time collaborative environment for Jupyter Notebooks, LaTeX documents, and SageMath, scalable from individual users to large groups and classes!

GitHub Repository: better-data-science/TensorFlow
Path: blob/main/006_Callbacks.ipynb
Views: 47
Kernel: tf
import os import numpy as np import pandas as pd import warnings os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' warnings.filterwarnings('ignore') df = pd.read_csv('data/winequalityN.csv') df.sample(5)
from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler # Prepare the data df = df.dropna() df['is_white_wine'] = [1 if typ == 'white' else 0 for typ in df['type']] df['is_good_wine'] = [1 if quality >= 6 else 0 for quality in df['quality']] df.drop(['type', 'quality'], axis=1, inplace=True) # Train/test split X = df.drop('is_good_wine', axis=1) y = df['is_good_wine'] X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) # Scaling scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test)

Modelling

import tensorflow as tf tf.random.set_seed(42)
Init Plugin Init Graph Optimizer Init Kernel

Callbacks list

  • I like to declare it beforehand

ModelCheckpoint

  • It will save the model locally on the current epoch if it beats the performance on the previous one

  • The configuration below saves it to a hdf5 file in the following format:

    • <dir>/model-<epoch>-<accuracy>.hdf5

  • Model is saved only if the validation accuracy is higher than on the previous epoch

cb_checkpoint = tf.keras.callbacks.ModelCheckpoint( filepath='checkpoints/model-{epoch:02d}-{val_accuracy:.2f}.hdf5', monitor='val_accuracy', mode='max', save_best_only=True, verbose=1 )

ReduceLROnPlateau

  • Basically if a metric (validation loss) doesn't decrease for a number of epochs (10), reduce the learning rate

  • New learning rate = old learning rate * factor (0.1)

    • nlr = 0.01 * 0.1 = 0.001

  • You can also set the minimum learning rate below the model won't go

cb_reducelr = tf.keras.callbacks.ReduceLROnPlateau( monitor='val_loss', mode='min', factor=0.1, patience=10, verbose=1, min_lr=0.00001 )

EarlyStopping

  • If a metric (validation accuracy) doesn't increase by some minimum delta (0.001) for a given number of epochs (10) - kill the training process

cb_earlystop = tf.keras.callbacks.EarlyStopping( monitor='val_accuracy', mode='max', min_delta=0.001, patience=10, verbose=1 )

CSVLogger

  • Captures model training history and dumps it to a CSV file

  • Useful for analyzing the performance later

cb_csvlogger = tf.keras.callbacks.CSVLogger( filename='training_log.csv', separator=',', append=False )

  • For simplicity's sake we'll treat test set as a validation set

  • In real deep learning projects you'll want to have 3 sets: training, validation, and test

  • We'll tell the model to train for 1000 epochs, but the EarlyStopping callback will kill it way before

  • Specify callbacks in the fit() function

model = tf.keras.Sequential([ tf.keras.layers.Dense(64, activation='relu'), tf.keras.layers.Dense(64, activation='relu'), tf.keras.layers.Dense(64, activation='relu'), tf.keras.layers.Dense(1, activation='sigmoid') ]) model.compile( loss=tf.keras.losses.binary_crossentropy, optimizer=tf.keras.optimizers.Adam(), metrics=[tf.keras.metrics.BinaryAccuracy(name='accuracy')] ) history = model.fit( X_train_scaled, y_train, epochs=1000, validation_data=(X_test_scaled, y_test), callbacks=[cb_checkpoint, cb_reducelr, cb_earlystop, cb_csvlogger] )
Metal device set to: Apple M1 Epoch 1/1000 162/162 [==============================] - 1s 7ms/step - loss: 0.5375 - accuracy: 0.7306 - val_loss: 0.4913 - val_accuracy: 0.7649 Epoch 00001: val_accuracy improved from -inf to 0.76489, saving model to checkpoints/model-01-0.76.hdf5 Epoch 2/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.4982 - accuracy: 0.7580 - val_loss: 0.5027 - val_accuracy: 0.7401 Epoch 00002: val_accuracy did not improve from 0.76489 Epoch 3/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.4880 - accuracy: 0.7689 - val_loss: 0.4774 - val_accuracy: 0.7703 Epoch 00003: val_accuracy improved from 0.76489 to 0.77030, saving model to checkpoints/model-03-0.77.hdf5 Epoch 4/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.4739 - accuracy: 0.7749 - val_loss: 0.4809 - val_accuracy: 0.7564 Epoch 00004: val_accuracy did not improve from 0.77030 Epoch 5/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.4681 - accuracy: 0.7791 - val_loss: 0.4767 - val_accuracy: 0.7657 Epoch 00005: val_accuracy did not improve from 0.77030 Epoch 6/1000 162/162 [==============================] - 1s 7ms/step - loss: 0.4595 - accuracy: 0.7824 - val_loss: 0.4660 - val_accuracy: 0.7688 Epoch 00006: val_accuracy did not improve from 0.77030 Epoch 7/1000 162/162 [==============================] - 1s 7ms/step - loss: 0.4533 - accuracy: 0.7872 - val_loss: 0.4647 - val_accuracy: 0.7780 Epoch 00007: val_accuracy improved from 0.77030 to 0.77804, saving model to checkpoints/model-07-0.78.hdf5 Epoch 8/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.4464 - accuracy: 0.7894 - val_loss: 0.4621 - val_accuracy: 0.7804 Epoch 00008: val_accuracy improved from 0.77804 to 0.78036, saving model to checkpoints/model-08-0.78.hdf5 Epoch 9/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.4398 - accuracy: 0.7956 - val_loss: 0.4605 - val_accuracy: 0.7780 Epoch 00009: val_accuracy did not improve from 0.78036 Epoch 10/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.4359 - accuracy: 0.7986 - val_loss: 0.4728 - val_accuracy: 0.7773 Epoch 00010: val_accuracy did not improve from 0.78036 Epoch 11/1000 162/162 [==============================] - 1s 7ms/step - loss: 0.4287 - accuracy: 0.8035 - val_loss: 0.4670 - val_accuracy: 0.7811 Epoch 00011: val_accuracy improved from 0.78036 to 0.78113, saving model to checkpoints/model-11-0.78.hdf5 Epoch 12/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.4218 - accuracy: 0.8095 - val_loss: 0.4602 - val_accuracy: 0.7842 Epoch 00012: val_accuracy improved from 0.78113 to 0.78422, saving model to checkpoints/model-12-0.78.hdf5 Epoch 13/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.4149 - accuracy: 0.8135 - val_loss: 0.4902 - val_accuracy: 0.7711 Epoch 00013: val_accuracy did not improve from 0.78422 Epoch 14/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.4150 - accuracy: 0.8064 - val_loss: 0.4626 - val_accuracy: 0.7796 Epoch 00014: val_accuracy did not improve from 0.78422 Epoch 15/1000 162/162 [==============================] - 1s 7ms/step - loss: 0.4054 - accuracy: 0.8164 - val_loss: 0.4704 - val_accuracy: 0.7858 Epoch 00015: val_accuracy improved from 0.78422 to 0.78577, saving model to checkpoints/model-15-0.79.hdf5 Epoch 16/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.4026 - accuracy: 0.8188 - val_loss: 0.4716 - val_accuracy: 0.7734 Epoch 00016: val_accuracy did not improve from 0.78577 Epoch 17/1000 162/162 [==============================] - 1s 7ms/step - loss: 0.3943 - accuracy: 0.8215 - val_loss: 0.4634 - val_accuracy: 0.7796 Epoch 00017: val_accuracy did not improve from 0.78577 Epoch 18/1000 162/162 [==============================] - 1s 7ms/step - loss: 0.3902 - accuracy: 0.8203 - val_loss: 0.4661 - val_accuracy: 0.7881 Epoch 00018: val_accuracy improved from 0.78577 to 0.78809, saving model to checkpoints/model-18-0.79.hdf5 Epoch 19/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.3837 - accuracy: 0.8267 - val_loss: 0.4740 - val_accuracy: 0.7819 Epoch 00019: val_accuracy did not improve from 0.78809 Epoch 20/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.3793 - accuracy: 0.8273 - val_loss: 0.4659 - val_accuracy: 0.7850 Epoch 00020: val_accuracy did not improve from 0.78809 Epoch 21/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.3738 - accuracy: 0.8338 - val_loss: 0.4632 - val_accuracy: 0.7873 Epoch 00021: val_accuracy did not improve from 0.78809 Epoch 22/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.3691 - accuracy: 0.8306 - val_loss: 0.4632 - val_accuracy: 0.7842 Epoch 00022: val_accuracy did not improve from 0.78809 Epoch 00022: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513. Epoch 23/1000 162/162 [==============================] - 1s 7ms/step - loss: 0.3395 - accuracy: 0.8516 - val_loss: 0.4615 - val_accuracy: 0.7920 Epoch 00023: val_accuracy improved from 0.78809 to 0.79196, saving model to checkpoints/model-23-0.79.hdf5 Epoch 24/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.3334 - accuracy: 0.8578 - val_loss: 0.4618 - val_accuracy: 0.7958 Epoch 00024: val_accuracy improved from 0.79196 to 0.79582, saving model to checkpoints/model-24-0.80.hdf5 Epoch 25/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.3311 - accuracy: 0.8586 - val_loss: 0.4614 - val_accuracy: 0.7981 Epoch 00025: val_accuracy improved from 0.79582 to 0.79814, saving model to checkpoints/model-25-0.80.hdf5 Epoch 26/1000 162/162 [==============================] - 1s 7ms/step - loss: 0.3290 - accuracy: 0.8602 - val_loss: 0.4648 - val_accuracy: 0.7966 Epoch 00026: val_accuracy did not improve from 0.79814 Epoch 27/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.3281 - accuracy: 0.8590 - val_loss: 0.4631 - val_accuracy: 0.7912 Epoch 00027: val_accuracy did not improve from 0.79814 Epoch 28/1000 162/162 [==============================] - 1s 7ms/step - loss: 0.3265 - accuracy: 0.8594 - val_loss: 0.4626 - val_accuracy: 0.7935 Epoch 00028: val_accuracy did not improve from 0.79814 Epoch 29/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.3252 - accuracy: 0.8627 - val_loss: 0.4637 - val_accuracy: 0.7958 Epoch 00029: val_accuracy did not improve from 0.79814 Epoch 30/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.3239 - accuracy: 0.8609 - val_loss: 0.4650 - val_accuracy: 0.7935 Epoch 00030: val_accuracy did not improve from 0.79814 Epoch 31/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.3230 - accuracy: 0.8623 - val_loss: 0.4667 - val_accuracy: 0.7943 Epoch 00031: val_accuracy did not improve from 0.79814 Epoch 32/1000 162/162 [==============================] - 1s 7ms/step - loss: 0.3218 - accuracy: 0.8634 - val_loss: 0.4705 - val_accuracy: 0.7920 Epoch 00032: val_accuracy did not improve from 0.79814 Epoch 00032: ReduceLROnPlateau reducing learning rate to 1.0000000474974514e-05. Epoch 33/1000 162/162 [==============================] - 1s 7ms/step - loss: 0.3187 - accuracy: 0.8613 - val_loss: 0.4664 - val_accuracy: 0.7951 Epoch 00033: val_accuracy did not improve from 0.79814 Epoch 34/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.3176 - accuracy: 0.8652 - val_loss: 0.4655 - val_accuracy: 0.7958 Epoch 00034: val_accuracy did not improve from 0.79814 Epoch 35/1000 162/162 [==============================] - 1s 6ms/step - loss: 0.3174 - accuracy: 0.8665 - val_loss: 0.4651 - val_accuracy: 0.7958 Epoch 00035: val_accuracy did not improve from 0.79814 Epoch 00035: early stopping

Final evaluation

  • You can now load the best model - it will be the one with the highest epoch number

best_model = tf.keras.models.load_model('checkpoints/model-25-0.80.hdf5')
  • Save yourself some time by calling predict_classes() instead of predict()

  • It assigns the classes automatically - you don't have to calculate them from probabilities

best_model_preds = np.ravel(best_model.predict_classes(X_test_scaled)) best_model_preds
array([1, 1, 0, ..., 1, 0, 1], dtype=int32)
  • Evaluate as you normally would

from sklearn.metrics import accuracy_score print(accuracy_score(y_test, best_model_preds))
0.7981438515081206