Accuracy not improving - image classification

Hi there! I am trying to train the classifier that consists of 9 labels. This is split into train, val and test. The value counts for each label in the training data is mentioned below. When I train the model even with a deep layered neural network, it doesn’t seem to change much. I have used Resnet 18-152, VGG16, alexnet, but it doesn’t give a high accuracy. I also don’t understand why each epoch’s accuracy is unpredictable. More number of epochs doesn’t seem to bring a big change in the accuracy either. I have also tried tuning the parameters in the optimizer but it doesn’t help as well. I am creating a flask application using this model but I can’t continue without finding the issue.

Epoch 0/24
----------
train Loss: 1.9129 Acc: 0.2992
val Loss: 1.2543 Acc: 0.6109

Epoch 1/24
----------
train Loss: 1.7522 Acc: 0.3500
val Loss: 1.0652 Acc: 0.6610

Epoch 2/24
----------
train Loss: 1.6867 Acc: 0.3759
val Loss: 1.0986 Acc: 0.6471

Epoch 3/24
----------
train Loss: 1.6434 Acc: 0.3972
val Loss: 2.9469 Acc: 0.5304

Epoch 4/24
----------
train Loss: 1.6174 Acc: 0.3987
val Loss: 1.6511 Acc: 0.4928

Epoch 5/24
----------
train Loss: 1.5869 Acc: 0.4187
val Loss: 1.2147 Acc: 0.6512

Epoch 6/24
----------
train Loss: 1.5569 Acc: 0.4254
val Loss: 1.1117 Acc: 0.6646

Epoch 7/24
----------
train Loss: 1.5612 Acc: 0.4304
val Loss: 0.9236 Acc: 0.6832

Epoch 8/24
----------
train Loss: 1.5131 Acc: 0.4396
val Loss: 1.0802 Acc: 0.6827

Epoch 9/24
----------
train Loss: 1.5151 Acc: 0.4396
val Loss: 1.0663 Acc: 0.6914

Epoch 10/24
----------
train Loss: 1.4907 Acc: 0.4433
val Loss: 0.9161 Acc: 0.6904

Epoch 11/24
----------
train Loss: 1.4973 Acc: 0.4532
val Loss: 1.3146 Acc: 0.6558

Epoch 12/24
----------
train Loss: 1.4802 Acc: 0.4537
val Loss: 1.0379 Acc: 0.6739

Epoch 13/24
----------
train Loss: 1.4678 Acc: 0.4531
val Loss: 1.1062 Acc: 0.6481

Epoch 14/24
----------
train Loss: 1.4542 Acc: 0.4602
val Loss: 1.0109 Acc: 0.6899

Epoch 15/24
----------
train Loss: 1.4302 Acc: 0.4687
val Loss: 0.8217 Acc: 0.7090

Epoch 16/24
----------
train Loss: 1.4087 Acc: 0.4782
val Loss: 0.9820 Acc: 0.6889

Epoch 17/24
----------
train Loss: 1.4056 Acc: 0.4847
val Loss: 1.0069 Acc: 0.6749

Epoch 18/24
----------
train Loss: 1.4144 Acc: 0.4670
val Loss: 0.8460 Acc: 0.7054

Epoch 19/24
----------
train Loss: 1.3993 Acc: 0.4872
val Loss: 0.9765 Acc: 0.6889

Epoch 20/24
----------
train Loss: 1.3896 Acc: 0.4889
val Loss: 0.8044 Acc: 0.7198

Epoch 21/24
----------
train Loss: 1.3591 Acc: 0.4913
val Loss: 0.8682 Acc: 0.6956

Epoch 22/24
----------
train Loss: 1.3473 Acc: 0.5065
val Loss: 0.8547 Acc: 0.7007

Epoch 23/24
----------
train Loss: 1.3593 Acc: 0.4918
val Loss: 0.9752 Acc: 0.6914

Epoch 24/24
----------
train Loss: 1.3354 Acc: 0.5024
val Loss: 0.8254 Acc: 0.7059

Training complete in 46m 18s
Best val Acc: 0.719814

Here is my classification report

              precision    recall  f1-score   support

          AK       0.13      0.65      0.22       161
         BCC       0.39      0.34      0.37       500
         BKL       0.14      0.09      0.11       393
          DF       1.00      0.10      0.18        50
         SCC       0.13      0.24      0.17        98
        VASC       0.60      0.48      0.53        73
    melanoma       0.45      0.63      0.53       762
       nevus       0.49      0.38      0.43       901
     unknown       0.98      0.81      0.89      3126

    accuracy                           0.62      6064
   macro avg       0.48      0.41      0.38      6064
weighted avg       0.70      0.62      0.64      6064

confusion matrix
image
value counts of each label in the training data

melanoma - 883
nevus = 889
bcc = 880
ak = 867
SCC = 628
unknown = 773
vasc = 253
bkl = 239

and here is my code

import numpy as np
import time
import copy
import os
import torch
import torch.optim as optim
import torch.nn as nn
import torchvision
import matplotlib.pyplot as plt
from torch.optim import lr_scheduler
from torchvision import datasets, models, transforms
import time
from tqdm import tqdm
import torch.nn.functional as F
from torch.nn.functional import relu

transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'test': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])
}

# generate the dataset and the loader

data_dir = '../input/data-final/data'

# image dataset are the ones that help in getting the image folder and the augmentation happening
image_dataset = {x: datasets.ImageFolder(os.path.join(data_dir, x), transform=transforms[x]) for x in ['train', 'val', 'test']}

# shuffling and loading the data to become the torch loader

dataloaders = {x: torch.utils.data.DataLoader(image_dataset[x], batch_size = 16, shuffle = True, num_workers = 4) for x in ['train', 'val', 'test']}

# getting the class names
class_names = image_dataset['train'].classes

# size of the data
data_size = {x: len(image_dataset[x]) for x in ['train', 'val', 'test']}


device = torch.device('cuda')
# create a model
def create_model(n_classes):
    model = models.resnet152(pretrained=True)
    n_features = model.fc.in_features
    model.fc = nn.Linear(n_features, n_classes)
    return model.to(device)

base_model = create_model(n_classes = 9)

# Load the model (Download only at first time loading the model)


# Define the function to train the model
def train_model(model, criterion, optimizer, num_epochs=25):
    since = time.time()
    
    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0
    
    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)
        
        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()
            else:
                model.eval()
                
            running_loss = 0.0
            running_corrects = 0
            
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)
                
                optimizer.zero_grad()
                
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)
                    
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()
                        
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
                

                
            epoch_loss = running_loss / data_size[phase]
            epoch_acc = running_corrects.double() / data_size[phase]
            
            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))
            
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())
        print()
    
    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))
    
    model.load_state_dict(best_model_wts)
    return model

# Set the requires_grad on each parameter to false, 
# so it will not calculate the gradients




# Set the new final layer for our new dataset




# Enable GPU for the model



# Set the loss function
criterion = nn.CrossEntropyLoss()

# Set the optimizer and the scheduler to update the weights, and train the model.

optimizer_conv = optim.Adam(base_model.parameters(), lr=0.001)

model_resnet = train_model(base_model, criterion, optimizer_conv, num_epochs=25)

Any support is appreciated.

Big Models can overfit and so it if not a gurantee a bigger model will always perform better.

I see you have tranforms (resize and centercrop) against valid and test set. I do not think you intend to do TTA. So, can you drop these two transforms and see how your model (ResNet18 for start) performs?

Hi Karthik, Thank you so much for the reply. As you said I have removed the augmentations from the phases and used Resnet 18 as the model. I used them as the labels seemed imbalanced. Now the validation accuracy increased to 83% from 71% . However the training accuracy is always at an average of 99%( it even showed 100% accuracy in a few epochs).The model is definitely overfitting at this point. How can I have a realistic training accuracy and increase the validation accuracy as well? Thanks.
This is my epoch history btw.

Epoch 0/69
----------
train Loss: 0.4579 Acc: 0.8418
val Loss: 0.7711 Acc: 0.7895

Epoch 1/69
----------
train Loss: 0.3891 Acc: 0.8604
val Loss: 0.7123 Acc: 0.8086

Epoch 2/69
----------
train Loss: 0.3274 Acc: 0.8912
val Loss: 1.1504 Acc: 0.7492

Epoch 3/69
----------
train Loss: 0.2924 Acc: 0.8968
val Loss: 0.9519 Acc: 0.8013

Epoch 4/69
----------
train Loss: 0.2063 Acc: 0.9292
val Loss: 1.3863 Acc: 0.7260

Epoch 5/69
----------
train Loss: 0.1951 Acc: 0.9304
val Loss: 1.0404 Acc: 0.7761

Epoch 6/69
----------
train Loss: 0.1559 Acc: 0.9453
val Loss: 0.8151 Acc: 0.8204

Epoch 7/69
----------
train Loss: 0.1608 Acc: 0.9455
val Loss: 1.9316 Acc: 0.7549

Epoch 8/69
----------
train Loss: 0.1283 Acc: 0.9557
val Loss: 1.5562 Acc: 0.7585

Epoch 9/69
----------
train Loss: 0.1162 Acc: 0.9616
val Loss: 1.6801 Acc: 0.7023

Epoch 10/69
----------
train Loss: 0.0999 Acc: 0.9665
val Loss: 1.0993 Acc: 0.8065

Epoch 11/69
----------
train Loss: 0.0774 Acc: 0.9742
val Loss: 1.1724 Acc: 0.7967

Epoch 12/69
----------
train Loss: 0.0864 Acc: 0.9698
val Loss: 1.2067 Acc: 0.8122

Epoch 13/69
----------
train Loss: 0.0795 Acc: 0.9738
val Loss: 1.0400 Acc: 0.8091

Epoch 14/69
----------
train Loss: 0.0600 Acc: 0.9796
val Loss: 1.8943 Acc: 0.7085

Epoch 15/69
----------
train Loss: 0.0581 Acc: 0.9804
val Loss: 1.3067 Acc: 0.7843

Epoch 16/69
----------
train Loss: 0.0624 Acc: 0.9802
val Loss: 1.3855 Acc: 0.8080

Epoch 17/69
----------
train Loss: 0.0624 Acc: 0.9796
val Loss: 1.0680 Acc: 0.8194

Epoch 18/69
----------
train Loss: 0.0443 Acc: 0.9864
val Loss: 1.2256 Acc: 0.8137

Epoch 19/69
----------
train Loss: 0.0242 Acc: 0.9908
val Loss: 1.5738 Acc: 0.8091

Epoch 20/69
----------
train Loss: 0.0314 Acc: 0.9915
val Loss: 1.3549 Acc: 0.8075

Epoch 21/69
----------
train Loss: 0.0345 Acc: 0.9888
val Loss: 1.3210 Acc: 0.8019

Epoch 22/69
----------
train Loss: 0.0364 Acc: 0.9881
val Loss: 1.7013 Acc: 0.7967

Epoch 23/69
----------
train Loss: 0.0312 Acc: 0.9899
val Loss: 1.5513 Acc: 0.8091

Epoch 24/69
----------
train Loss: 0.0579 Acc: 0.9804
val Loss: 1.4634 Acc: 0.8111

Epoch 25/69
----------
train Loss: 0.0345 Acc: 0.9899
val Loss: 1.1516 Acc: 0.8070

Epoch 26/69
----------
train Loss: 0.0225 Acc: 0.9924
val Loss: 1.6052 Acc: 0.7962

Epoch 27/69
----------
train Loss: 0.0102 Acc: 0.9967
val Loss: 1.3944 Acc: 0.8189

Epoch 28/69
----------
train Loss: 0.0064 Acc: 0.9979
val Loss: 1.7504 Acc: 0.8127

Epoch 29/69
----------
train Loss: 0.0315 Acc: 0.9910
val Loss: 1.4341 Acc: 0.8204

Epoch 30/69
----------
train Loss: 0.0270 Acc: 0.9910
val Loss: 1.5566 Acc: 0.8013

Epoch 31/69
----------
train Loss: 0.0095 Acc: 0.9972
val Loss: 1.2792 Acc: 0.8344

Epoch 32/69
----------
train Loss: 0.0094 Acc: 0.9968
val Loss: 1.7946 Acc: 0.7879

Epoch 33/69
----------
train Loss: 0.0255 Acc: 0.9911
val Loss: 1.4167 Acc: 0.8148

Epoch 34/69
----------
train Loss: 0.0244 Acc: 0.9935
val Loss: 1.5633 Acc: 0.7585

Epoch 35/69
----------
train Loss: 0.0110 Acc: 0.9972
val Loss: 1.4525 Acc: 0.8179

Epoch 36/69
----------
train Loss: 0.0036 Acc: 0.9987
val Loss: 1.3777 Acc: 0.8225

Epoch 37/69
----------
train Loss: 0.0014 Acc: 0.9997
val Loss: 1.3799 Acc: 0.8204

Epoch 38/69
----------
train Loss: 0.0039 Acc: 0.9986
val Loss: 1.4773 Acc: 0.8065

Epoch 39/69
----------
train Loss: 0.0060 Acc: 0.9979
val Loss: 1.7012 Acc: 0.8060

Epoch 40/69
----------
train Loss: 0.0060 Acc: 0.9981
val Loss: 1.5426 Acc: 0.8127

Epoch 41/69
----------
train Loss: 0.0042 Acc: 0.9989
val Loss: 1.4538 Acc: 0.8189

Epoch 42/69
----------
train Loss: 0.0021 Acc: 0.9994
val Loss: 1.5516 Acc: 0.8220

Epoch 43/69
----------
train Loss: 0.0067 Acc: 0.9975
val Loss: 1.7034 Acc: 0.8039

Epoch 44/69
----------
train Loss: 0.0182 Acc: 0.9946
val Loss: 1.4298 Acc: 0.8277

Epoch 45/69
----------
train Loss: 0.0342 Acc: 0.9889
val Loss: 1.3819 Acc: 0.8132

Epoch 46/69
----------
train Loss: 0.0155 Acc: 0.9953
val Loss: 1.4299 Acc: 0.8142

Epoch 47/69
----------
train Loss: 0.0094 Acc: 0.9968
val Loss: 1.5808 Acc: 0.7993

Epoch 48/69
----------
train Loss: 0.0087 Acc: 0.9973
val Loss: 1.2893 Acc: 0.8323

Epoch 49/69
----------
train Loss: 0.0068 Acc: 0.9978
val Loss: 1.6876 Acc: 0.7951

Epoch 50/69
----------
train Loss: 0.0076 Acc: 0.9981
val Loss: 1.2886 Acc: 0.8235

Epoch 51/69
----------
train Loss: 0.0183 Acc: 0.9938
val Loss: 1.4195 Acc: 0.8199

Epoch 52/69
----------
train Loss: 0.0167 Acc: 0.9951
val Loss: 1.2933 Acc: 0.8209

Epoch 53/69
----------
train Loss: 0.0055 Acc: 0.9984
val Loss: 1.4341 Acc: 0.8179

Epoch 54/69
----------
train Loss: 0.0024 Acc: 0.9997
val Loss: 1.5026 Acc: 0.8148

Epoch 55/69
----------
train Loss: 0.0026 Acc: 0.9992
val Loss: 1.6046 Acc: 0.8173

Epoch 56/69
----------
train Loss: 0.0044 Acc: 0.9981
val Loss: 1.3364 Acc: 0.8251

Epoch 57/69
----------
train Loss: 0.0016 Acc: 0.9995
val Loss: 1.4984 Acc: 0.8220

Epoch 58/69
----------
train Loss: 0.0005 Acc: 1.0000
val Loss: 1.5610 Acc: 0.8153

Epoch 59/69
----------
train Loss: 0.0017 Acc: 0.9995
val Loss: 1.3520 Acc: 0.8364

Epoch 60/69
----------
train Loss: 0.0015 Acc: 0.9994
val Loss: 1.5953 Acc: 0.8034

Epoch 61/69
----------
train Loss: 0.0007 Acc: 1.0000
val Loss: 1.6250 Acc: 0.8142

Epoch 62/69
----------
train Loss: 0.0005 Acc: 1.0000
val Loss: 1.5463 Acc: 0.8117

Epoch 63/69
----------
train Loss: 0.0012 Acc: 0.9997
val Loss: 1.5688 Acc: 0.8199

Epoch 64/69
----------
train Loss: 0.0008 Acc: 0.9998
val Loss: 1.4750 Acc: 0.8225

Epoch 65/69
----------
train Loss: 0.0006 Acc: 0.9997
val Loss: 1.6174 Acc: 0.8086

Epoch 66/69
----------
train Loss: 0.0003 Acc: 1.0000
val Loss: 1.6037 Acc: 0.8132

Epoch 67/69
----------
train Loss: 0.0009 Acc: 0.9997
val Loss: 1.5039 Acc: 0.8282

Epoch 68/69
----------
train Loss: 0.0041 Acc: 0.9992
val Loss: 1.5608 Acc: 0.8199

Epoch 69/69
----------
train Loss: 0.0008 Acc: 0.9998
val Loss: 1.5780 Acc: 0.8158

Training complete in 63m 10s
Best val Acc: 0.836429

and this is the confusion matrix
image

Since you mention it is a imbalanced dataset, I would not focus on accuracy but other form of loss and metrics.

During training,

  1. There is WeightedRandomSampler which you can use (please search in this forum and you will find answered queries). Or,
  2. You are using CrossEntropy loss. This accepts weights for each class. Look here.

For metrics, you could look at F1 score

You can explore the above and see what works. Let us know.