Transfer Learning with MobileNetV2

Welcome to this week's assignment, where you'll be using transfer learning on a pre-trained CNN to build an Alpaca/Not Alpaca classifier!

A pre-trained model is a network that's already been trained on a large dataset and saved, which allows you to use it to customize your own model cheaply and efficiently. The one you'll be using, MobileNetV2, was designed to provide fast and computationally efficient performance. It's been pre-trained on ImageNet, a dataset containing over 14 million images and 1000 classes.

By the end of this assignment, you will be able to:

• Create a dataset from a directory

• Preprocess and augment data using the Sequential API

• Adapt a pretrained model to new data and train a classifier using the Functional API and MobileNet

• Fine-tune a classifier's final layers to improve accuracy

Important Note on Submission to the AutoGrader

Before submitting your assignment to the AutoGrader, please make sure you are not doing the following:

1. You have not added any extra print statement(s) in the assignment.

2. You have not added any extra code cell(s) in the assignment.

3. You have not changed any of the function parameters.

4. You are not using any global variables inside your graded exercises. Unless specifically instructed to do so, please refrain from it and use the local variables instead.

5. You are not changing the assignment code where it is not required, like creating extra variables.

If you do any of the following, you will get something like, Grader not found (or similarly unexpected) error upon submitting your assignment. Before asking for help/debugging the errors in your assignment, check for these first. If this is the case, and you don't remember the changes you have made, you can get a fresh copy of the assignment by following these instructions.

Table of Content

• 1 - Packages

  • 1.1 Create the Dataset and Split it into Training and Validation Sets

• 2 - Preprocess and Augment Training Data

  • Exercise 1 - data_augmenter

• 3 - Using MobileNetV2 for Transfer Learning

  • 3.1 - Inside a MobileNetV2 Convolutional Building Block

  • 3.2 - Layer Freezing with the Functional API

    • Exercise 2 - alpaca_model

  • 3.3 - Fine-tuning the Model

    • Exercise 3

1 - Packages

1.1 Create the Dataset and Split it into Training and Validation Sets

When training and evaluating deep learning models in Keras, generating a dataset from image files stored on disk is simple and fast. Call image_data_set_from_directory() to read from the directory and create both training and validation datasets.

If you're specifying a validation split, you'll also need to specify the subset for each portion. Just set the training set to subset='training' and the validation set to subset='validation'.

You'll also set your seeds to match each other, so your training and validation sets don't overlap. 😃

Now let's take a look at some of the images from the training set:

Note: The original dataset has some mislabelled images in it as well.

2 - Preprocess and Augment Training Data

You may have encountered dataset.prefetch in a previous TensorFlow assignment, as an important extra step in data preprocessing.

Using prefetch() prevents a memory bottleneck that can occur when reading from disk. It sets aside some data and keeps it ready for when it's needed, by creating a source dataset from your input data, applying a transformation to preprocess it, then iterating over the dataset one element at a time. Because the iteration is streaming, the data doesn't need to fit into memory.

You can set the number of elements to prefetch manually, or you can use tf.data.experimental.AUTOTUNE to choose the parameters automatically. Autotune prompts tf.data to tune that value dynamically at runtime, by tracking the time spent in each operation and feeding those times into an optimization algorithm. The optimization algorithm tries to find the best allocation of its CPU budget across all tunable operations.

To increase diversity in the training set and help your model learn the data better, it's standard practice to augment the images by transforming them, i.e., randomly flipping and rotating them. Keras' Sequential API offers a straightforward method for these kinds of data augmentations, with built-in, customizable preprocessing layers. These layers are saved with the rest of your model and can be re-used later. Ahh, so convenient!

As always, you're invited to read the official docs, which you can find for data augmentation here.

Exercise 1 - data_augmenter

Implement a function for data augmentation. Use a Sequential keras model composed of 2 layers:

• RandomFlip('horizontal')

• RandomRotation(0.2)

Take a look at how an image from the training set has been augmented with simple transformations:

From one cute animal, to 9 variations of that cute animal, in three lines of code. Now your model has a lot more to learn from.

Next, you'll apply your first tool from the MobileNet application in TensorFlow, to normalize your input. Since you're using a pre-trained model that was trained on the normalization values [-1,1], it's best practice to reuse that standard with tf.keras.applications.mobilenet_v2.preprocess_input.

What you should remember:

• When calling image_data_set_from_directory(), specify the train/val subsets and match the seeds to prevent overlap

• Use prefetch() to prevent memory bottlenecks when reading from disk

• Give your model more to learn from with simple data augmentations like rotation and flipping.

• When using a pretrained model, it's best to reuse the weights it was trained on.

3 - Using MobileNetV2 for Transfer Learning

MobileNetV2 was trained on ImageNet and is optimized to run on mobile and other low-power applications. It's 155 layers deep (just in case you felt the urge to plot the model yourself, prepare for a long journey!) and very efficient for object detection and image segmentation tasks, as well as classification tasks like this one. The architecture has three defining characteristics:

• Depthwise separable convolutions

• Thin input and output bottlenecks between layers

• Shortcut connections between bottleneck layers

3.1 - Inside a MobileNetV2 Convolutional Building Block

MobileNetV2 uses depthwise separable convolutions as efficient building blocks. Traditional convolutions are often very resource-intensive, and depthwise separable convolutions are able to reduce the number of trainable parameters and operations and also speed up convolutions in two steps:

1. The first step calculates an intermediate result by convolving on each of the channels independently. This is the depthwise convolution.

2. In the second step, another convolution merges the outputs of the previous step into one. This gets a single result from a single feature at a time, and then is applied to all the filters in the output layer. This is the pointwise convolution, or: Shape of the depthwise convolution X Number of filters.

Figure 1 : MobileNetV2 Architecture
This diagram was inspired by the original seen here.

Each block consists of an inverted residual structure with a bottleneck at each end. These bottlenecks encode the intermediate inputs and outputs in a low dimensional space, and prevent non-linearities from destroying important information.

The shortcut connections, which are similar to the ones in traditional residual networks, serve the same purpose of speeding up training and improving predictions. These connections skip over the intermediate convolutions and connect the bottleneck layers.

Let's try to train your base model using all the layers from the pretrained model.

Similarly to how you reused the pretrained normalization values MobileNetV2 was trained on, you'll also load the pretrained weights from ImageNet by specifying weights='imagenet'.

Print the model summary below to see all the model's layers, the shapes of their outputs, and the total number of parameters, trainable and non-trainable.

Note the last 2 layers here. They are the so called top layers, and they are responsible of the classification in the model

Notice some of the layers in the summary like Conv2D and DepthwiseConv2D and how they follow the progression of expansion to depthwise convolution to projection. In combination with BatchNormalization and ReLU, these make up the bottleneck layers mentioned earlier.

What you should remember:

• MobileNetV2's unique features are:

  • Depthwise separable convolutions that provide lightweight feature filtering and creation

  • Input and output bottlenecks that preserve important information on either end of the block

• Depthwise separable convolutions deal with both spatial and depth (number of channels) dimensions

Next, choose the first batch from the tensorflow dataset to use the images, and run it through the MobileNetV2 base model to test out the predictions on some of your images.

Now decode the predictions made by the model. Earlier, when you printed the shape of the batch, it would have returned (32, 1000). The number 32 refers to the batch size and 1000 refers to the 1000 classes the model was pretrained on. The predictions returned by the base model below follow this format:

First the class number, then a human-readable label, and last the probability of the image belonging to that class. You'll notice that there are two of these returned for each image in the batch - these the top two probabilities returned for that image.

Uh-oh. There's a whole lot of labels here, some of them hilariously wrong, but none of them say "alpaca."

This is because MobileNet pretrained over ImageNet doesn't have the correct labels for alpacas, so when you use the full model, all you get is a bunch of incorrectly classified images.

Fortunately, you can delete the top layer, which contains all the classification labels, and create a new classification layer.

3.2 - Layer Freezing with the Functional API

In the next sections, you'll see how you can use a pretrained model to modify the classifier task so that it's able to recognize alpacas. You can achieve this in three steps:

1. Delete the top layer (the classification layer)

   • Set include_top in base_model as False

2. Add a new classifier layer

   • Train only one layer by freezing the rest of the network

   • As mentioned before, a single neuron is enough to solve a binary classification problem.

3. Freeze the base model and train the newly-created classifier layer

   • Set base model.trainable=False to avoid changing the weights and train only the new layer

   • Set training in base_model to False to avoid keeping track of statistics in the batch norm layer

Exercise 2 - alpaca_model

Create your new model using the data_augmentation function defined earlier.

The base learning rate has been set for you, so you can go ahead and compile the new model and run it for 5 epochs:

Plot the training and validation accuracy:

The results are ok, but could be better. Next, try some fine-tuning.

3.3 - Fine-tuning the Model

You could try fine-tuning the model by re-running the optimizer in the last layers to improve accuracy. When you use a smaller learning rate, you take smaller steps to adapt it a little more closely to the new data. In transfer learning, the way you achieve this is by unfreezing the layers at the end of the network, and then re-training your model on the final layers with a very low learning rate. Adapting your learning rate to go over these layers in smaller steps can yield more fine details - and higher accuracy.

The intuition for what's happening: when the network is in its earlier stages, it trains on low-level features, like edges. In the later layers, more complex, high-level features like wispy hair or pointy ears begin to emerge. For transfer learning, the low-level features can be kept the same, as they have common features for most images. When you add new data, you generally want the high-level features to adapt to it, which is rather like letting the network learn to detect features more related to your data, such as soft fur or big teeth.

To achieve this, just unfreeze the final layers and re-run the optimizer with a smaller learning rate, while keeping all the other layers frozen.

Where the final layers actually begin is a bit arbitrary, so feel free to play around with this number a bit. The important takeaway is that the later layers are the part of your network that contain the fine details (pointy ears, hairy tails) that are more specific to your problem.

First, unfreeze the base model by setting base_model.trainable=True, set a layer to fine-tune from, then re-freeze all the layers before it. Run it again for another few epochs, and see if your accuracy improved!

Exercise 3

Ahhh, quite an improvement! A little fine-tuning can really go a long way.

What you should remember:

• To adapt the classifier to new data: Delete the top layer, add a new classification layer, and train only on that layer

• When freezing layers, avoid keeping track of statistics (like in the batch normalization layer)

• Fine-tune the final layers of your model to capture high-level details near the end of the network and potentially improve accuracy

Congratulations!

You've completed this assignment on transfer learning and fine-tuning. Here's a quick recap of all you just accomplished:

• Created a dataset from a directory

• Augmented data with the Sequential API

• Adapted a pretrained model to new data with the Functional API and MobileNetV2

• Fine-tuned the classifier's final layers and boosted the model's accuracy

That's awesome!