Confusion matrix for valiation data

vision
1

Hi, I want to plot a confusion matrix for validation set. The code for my training is given below. Which is taken from pytorch example. During training phase it also computes and show the validation loss and accuracy as well as training loss and accuracy. But I also want to see the confusion matrix for validation set. How can I do it ?

def train_model(model, criterion, optimizer, scheduler, num_epochs):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch +1, num_epochs))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'valid']:
            if phase == 'train':
                scheduler.step()
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()
                    

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            #For graph generation
            if phase == "train":
                train_loss.append(running_loss/dataset_sizes[phase])
                train_acc.append(running_corrects.double() / dataset_sizes[phase])
                epoch_counter_train.append(epoch)
            if phase == "valid":
                val_loss.append(running_loss/ dataset_sizes[phase])
                val_acc.append(running_corrects.double() / dataset_sizes[phase])
                epoch_counter_val.append(epoch)

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            #for printing        
            if phase == "train":    
                epoch_loss = running_loss / dataset_sizes[phase]
                epoch_acc = running_corrects.double() / dataset_sizes[phase]
            if phase == "valid":    
                epoch_loss = running_loss / dataset_sizes[phase]
                epoch_acc = running_corrects.double() / dataset_sizes[phase]
            
            
            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the best model
            if phase == 'valid' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    #save the  best model
    torch.save(model.load_state_dict(best_model_wts), savePath+'bestModel.pth')
    return model

Now , I know how to plot confusion matrix for my test set too, like

with torch.no_grad():
    for i, (inputs, labels) in enumerate(dataloaders['test']):
        inputs = inputs.to(device)
        labels = labels.to(device)
        outputs = model_ft(inputs)
        _, predicted = torch.max(outputs, 1)
        confusion_matrix.add(predicted, labels)

But I am confused how to add the code snippet:
confusion_matrix.add(predicted, labels)
add where to incorporate the line in my training function. If I add it in the training function after the line

if phase == "valid":  
   confusion_matrix.add(preds, labels)

It seems, the result is not correct if I plot it.
So, Can you please suggest how to get the confusion matrix on validation set? Should I have to do it in my training function where I am both training and validating or just like the test set, outside training function, looping through validation set again ?

2

Hi @Rosa,
I think you can do it in your training function. This snippet should work:

confusion_matrix = torch.zeros(nb_labels, nb_labels)
for inputs, labels in dataloaders[phase]:
       inputs = inputs.to(device)
       labels = labels.to(device)
       optimizer.zero_grad()

       with torch.set_grad_enabled(phase == 'train'):
              outputs = model(inputs)
              _, preds = torch.max(outputs, 1)
              loss = criterion(outputs, labels)
              ....

              if phase == 'valid':
                      for l, p in zip(labels.view(-1), preds.view(-1)):
                      confusion_matrix[l.long(), p.long()] += 1
print(confusion_matrix)
1 Like
3

I am using this function confusion matrix. and getting this
Confusion Matirx
[[16 0 1 0 8 0 0]
[ 0 15 0 0 1 0 0]
[ 1 1 6 0 1 0 0]
[ 1 0 0 9 2 0 1]
[ 3 0 0 0 11 0 0]
[ 0 3 0 0 0 9 0]
[ 0 0 0 0 0 1 14]]

and it says accuracy score is 0.76923. It has 7 classes, Total number of samples are 104 So the overall accuracy is

Is it called the Weighted accuracy or unweighted accuracy?
And for other , we need to do the manual calculation?

4

Got it ,It is weighted , ie overall accuracy. In case of unweighted accuracy , equal weightage is given to each class.