Save and load models

snip3r77 · April 13, 2020, 5:42pm

udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part 6 - Saving and Loading Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Saving and Loading Models\n",
    "\n",
    "In this notebook, I'll show you how to save and load models with PyTorch. This is important because you'll often want to load previously trained models to use in making predictions or to continue training on new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",

This file has been truncated. show original

Question

So when we save the model and if we decided to tweak the hidden layers, we can just adjust the hidden layers while using the weights from model.state_dict() ? ( the bunch of codes towards the end )
Do we usually use this utility script to create a new NN config?
https://github.com/udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/helper.py

Thanks

ptrblck · April 14, 2020, 2:06am

The model parameters should match the stored parameters in the state_dict. Otherwise you’ll get a size mismatch error. You could store the original shapes in the checkpoint, if you think it’s convenient.
You could adapt the code to your use case, but note that the code hasn’t been updated in a while and still uses Variables, which were deprecated in PyTorch 0.4.

snip3r77 · April 14, 2020, 2:43am

def load_checkpoint(filepath):
    checkpoint = torch.load(filepath)
    model = fc_model.Network(checkpoint['input_size'],
                             checkpoint['output_size'],
                             checkpoint['hidden_layers'])
    model.load_state_dict(checkpoint['state_dict'])
    
    return model

Thank you for the reply.

So basically , we can re-use the weights but we need to change the architecture based on this function

What’s the difference if we re-train again when we define the network? In what way we’re re-using
checkpoint['state_dict'] and how it differs with transfer learning?

side track,

I’ve been going from coursera deeplearning.ai , fast.ai and now udacity’s pytorch. It seems like I prefer pytorch ( or fast.ai ) and I love the community here.

My question is how do I get used to the syntax
( i.e turn off gradient and dropout during inferencing ).
Thank you for the reply.

What I’m trying to say is there is a lot of steps that are consistent I think, so how does one remember these steps OR usually we do not need to remember ( just keep it in your notepad) and understand what it is trying to do. Because DL is definitely not as straight forward as sklearn’s .fit and .predict.

Thank you.

ptrblck · April 14, 2020, 7:26am

You shouldn’t change the architecture, but recreate the model using the same arguments as during the initial training.

Loading the state_dict is used to load all previously stored (and trained) parameters and buffers into the model. It’s the first step when you want to fine tune the model (transfer learning).

That’s great to hear!

Once you’ve created a training, validation and testing loop, you might realize that there are not that many things to remember.
Some common issues are, if you forget to call optimizer.zero_grad() (which still happens to me after some time using PyTorch ), or to forget to call model.train()/.eval().
That being said, I’m not sure if there are any gotchas, but let us know, if you think I’m missing something.

snip3r77 · April 18, 2020, 6:03pm

I’ve now reached ‘transfer learning’

It seems like the code is even easier as compared to training from scratch.

github.com

udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part 8 - Transfer Learning (Solution).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transfer Learning\n",
    "\n",
    "In this notebook, you'll learn how to use pre-trained networks to solved challenging problems in computer vision. Specifically, you'll use networks trained on [ImageNet](http://www.image-net.org/) [available from torchvision](http://pytorch.org/docs/0.3.0/torchvision/models.html). \n",
    "\n",
    "ImageNet is a massive dataset with over 1 million labeled images in 1000 categories. It's used to train deep neural networks using an architecture called convolutional layers. I'm not going to get into the details of convolutional networks here, but if you want to learn more about them, please [watch this](https://www.youtube.com/watch?v=2-Ol7ZB0MmU).\n",
    "\n",
    "Once trained, these models work astonishingly well as feature detectors for images they weren't trained on. Using a pre-trained network on images not in the training set is called transfer learning. Here we'll use transfer learning to train a network that can classify our cat and dog photos with near perfect accuracy.\n",
    "\n",
    "With `torchvision.models` you can download these pre-trained networks and use them in your applications. We'll include `models` in our imports now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,

This file has been truncated. show original

Some questions

My batch size is 64 and if I print every 5 steps.
With the following 4 steps below, does it mean I have already gone through 4x5steps = 20 batches? Or 20 x 64 = 1280 images?

Epoch 1/1..Train loss: 2.411.. Test loss: 0.593.. Test accuracy: 0.702
Epoch 1/1..Train loss: 1.049.. Test loss: 0.317.. Test accuracy: 0.873
Epoch 1/1..Train loss: 0.433.. Test loss: 0.191.. Test accuracy: 0.914
Epoch 1/1..Train loss: 0.302.. Test loss: 0.105.. Test accuracy: 0.963

I think I ran the full 1 epoch and the improvement is 1 or 2%. Do you usually suggest early stopping?
If I run this using fast.ai , will it will be significantly faster using the LR-finder ? Is there a similar one for PyTorch? I think Pytorch Lightning already released recently.
So usually how does one tweak the hyperparameter for transfer learning assuming I only obtain 75% instead of 96%?

Thanks

ptrblck · April 18, 2020, 11:28pm

Each iteration uses the specified batch size, so your calculation should be correct.
The accuracy improvement in the first iteration seems to come close to 26% in the first steps or are you referring to a later epoch? If you don’t see much improvement, I would suggest to stop the training.
This might be the case. FastAI uses a lot of utility functions and good practices to achieve really good results in less training time. I’m not sure which higher-level library has implemented the lr finder.
I would recommend to take a look at some FastAI lectures, as @jphoward explains a lot of techniques to properly fine tune a model. Also, you could use a Kaggle competition to try our some techniques and learn from others during the competition.

snip3r77 · April 23, 2020, 3:26pm

Hi,

I’m just wondering if SubsetRandomSampler include stratify whereby % of classes are consistent in train and valid.

Last but not least, is this the usual way to split 80/20 between train and test(codewise) or is there a train-test-split( apologies as I’m looking from the sklearn angle)? Thanks

ptrblck · April 23, 2020, 7:48pm

No, SubsetRandomSampler will just randomly shuffle the passed indices.

This would be a valid way, but I’m often using sklearn methods, in case I need to create stratified splits etc. If you are OK with adding sklearn as a dependency to your script, I would just use these methods.

snip3r77 · May 7, 2020, 1:16pm

so OpenCV is like an utility library to assist PyTorch in manipulating image?
I’m sure there is a few of such library. Will it be ok if I just still to one or is there a better library than OpenCV?

At the kernel level , when we apply a -ve to the pixel it will turn it to white(1) if it’s dark (255) and vice-versa ?

Thanks.

ptrblck · May 8, 2020, 4:03am

You can use every library you like during data loading and processing, as long as you don’t need to use this operation inside the model and need Autograd to calculate the backward pass through it.
torchvision uses PIL as the default image library, but you can of course use OpenCV.
Note that the color format of OpenCV is BGR, while PIL and thus all pretrained models in torchvision use RGB, so you should transform the color channels if you want to feed these images to a pretrained model.

A -1 will invert the pixel values, so dark colors in a grayscale image should get bright and vice versa, yes.

snip3r77 · May 10, 2020, 3:48am

So basically we’re doing an .unsqueeze(1) is because we wanted to match the shape. It’s grayscale hence we need to add the channel at axis =1 ?

BTW, I’m doing Udacity’s Pytorch course I think I’m now starting to feel uncomfortable when we reach CNN because there’s not much practice.

Also, usually when I code in Python, for things that I do not understand I can break it down piece by piece to debug and understand and for Pytorch I have ‘debugging skills’ issue. The codes are bundled and I can’t run step by step. Is there a tutorial on this?

Appreciate your help. Thanks.

More details

ptrblck · May 10, 2020, 6:42am

It depends, what you want to do with this tensor.
If you want to pass it to a 2D CNN, then you would have to provide an input shape of [batch_size, channels, height, width]. Since the channel dimension might be missing, you would have to unsqueeze it.

Could you run the cells separately for debugging purposes?
I’m not familiar with the course, but where are you stuck?

snip3r77 · May 12, 2020, 12:45pm

I have not start coding yet, I will bring this up when start coding CNN. Thanks

CNN Question

#Input is 130 x 130 x RGB
nn.Conv2d(3, 10, 3) <- line 1
nn.MaxPool2d(4, 4) <- line 2 
nn.Conv2d(10, 20, 5, padding=2) <- line 3
nn.MaxPool2d(2, 2) <- line 4

so basically at line 1 the input depth is 3 and the output is 10
and we use this formula

( input_depth + 2*pad - kernel )/ stride + 1

to determine the image size

example 130 - 3 / 1 + 1 = 128

From what I read the depth will be increasing ( from 3 to 20 ) and the image size will reduce ( divide by 4 and 2 )
Image will reduce due to MaxPool downscaling.

So when we see nn.MaxPool2d(4, 4) , the image will be divided by 4 . Is there a chance that the tuple of (4,4) will be a different value ?

So at line 2 will shrink by factor of 4 and at line 4 will shrink by a factor 2 .

Side tack
Also, if we want to maintain the image size with known kernel size and use padding to compensate we just use the following formula right?

( input_depth + 2*pad - kernel )/ stride + 1 = input_depth then re-arrange to find pad unknown

ptrblck · May 12, 2020, 10:49pm

The output shape formula uses the input height and width, not the depth.
Also some minor errors are in the formula, i.e. the dilation is missing as well as some constant factors.
Have a look at the docs for the complete formula.

Yes, this conv uses 3 input channels and 10 filters, which create one output channel each.

Yes, that would be the case.

I don’t understand this question. You could pass non-square kernels, e.g. (4, 2), if you want.

Yeah, but with the formula from the docs.

snip3r77 · May 13, 2020, 1:30am

Thanks for the prompt reply.

I got this

( input_depth + 2*pad - kernel )/ stride + 1 = input_depth

when I was going through some of Andrew Ng’s video.
I think he is trying to teach us the intuition but I guess there are more parameters like
dilation that I’m not exposed to.

I understood that (4,4) or (2,2) is reducing by a factor of 4 or 2 ?
Is it uncommon to use non square kernels?

So do we usually hand calculate these outputs or do we leave it for Pytorch to do so? Or do we need to understand some of these things under hood but let Pytorch to do the heavy lifting.

Cheers.

ptrblck · May 13, 2020, 1:58am

Yes, the spatial reduction would be 4x and 2x for these kernel shapes and strides.
It might be uncommon to use non-square kernels, but this doesn’t mean that you shouldn’t use them in your model, if you think they make sense for whatever reason.

You could calculate it manually, but for complicated use cases with an uncommon stride, dilation etc., I just print the output shape and adapt the parameters to match my desired output.

snip3r77 · May 13, 2020, 4:18pm

Questions

I manually calculated all the conv2d values
and also the FC1 and FC2 dimensions. Is there a way that it can be calculated by pytorch?

**Coincidentally, here with the use of padding 1, input equates to output before Pooling, hence
it’s slightly easier over here.

At line 41, is -1 referring to the batch size since we converted the rest to 4W * 4H * 64 final depth ?
I understand that I did a resnet transfer learning before and it can achieve >90% but when I use CNN, I only managed to obtain ~70% based on this network. Conceptually would it be better to just use transfer learning instead OR this is just an ‘entry level’ CNN that will lead to a much better one that may or may not be better than transfer learning?

Thanks.

ptrblck · May 14, 2020, 5:29am

There might be utility functions in the wild, but I would recommend to use a dummy input tensor, print all necessary activation shapes, and set the input features of linear layers in this way.
Since each forward pass is dynamic in PyTorch and the computation graph will be created during the forward pass, the shape inference is not straightforward to calculate in eager mode.
Yes, the activation will be flattened to [batch_size, 4*4*64]. I would generally recommend to use x.view(x.size(0), -1) to keep the batch size and run into shape mismatch errors instead of changing the batch size, which would be harder to debug.
That might be expected, as a pretrained ResNet is quite powerful for a variety of tasks.

snip3r77 · May 14, 2020, 3:45pm

Also I realise that if my input is RGB the input/yellow needs to be 3 and if it’s grayscale it needs to be 1. Slowly trying to provide intuition.

I don’t quite understand how to do this. As mentioned, I think this is the portion that I’m kinda lost ( I do not know how to debug/go to manual mode ). Is there any tutorial that explain on this?

But I did google and manage to find from Pytorch 60mins blitz from the documentation.

which I adapted to my code [51] . but I’m not sure how to proceed from here.

Also, I find this is pretty neat and with this I do not need to calculate the FC.

Thanks

ptrblck · May 15, 2020, 1:01am

You would add print(x.shape) statements into the forward method to all places, where you don’t know the activation shape or don’t want to calculate it manually for the current input shape.

The num_flatten_features don’t avoid calculating the in_features as 16*6*6, but help you to flatten the tensor. I would still recommend to use x.view(x.size(0), -1).