How to get the sensitivity and specificity of a dataset?

Guys, I am making a classifier using ResNet and I want to get the Sensitivity and specificity of the particular dataset. Right now I have accuracy, Train loss, and test loss. I have already studied from Wikipedia and YouTube, about True positive/negative, false negative/positive and know the formula. But I do not know how to implement the formula, like how to get true positive and false negatives. Here is my code -

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

model = models.resnet101(pretrained=True)

for param in model.parameters():
    param.requires_grad = True

model.fc = nn.Sequential(nn.Linear(2048, 1024),
                                 nn.ReLU(),
                                 nn.Dropout(0.4),
                                 nn.Linear(1024,4),
                                 nn.LogSoftmax(dim=1))

criterion = nn.NLLLoss()

optimizer = optim.Adam(model.fc.parameters(), lr = 0.001)

model.to(device);
                
epochs = 60
#steps = 0
#print_every = 5

for epoch in range(epochs):
  
  running_loss = 0
  model.train()
  for images, labels in dataloader_train:
    
    #steps += 1
    images, labels = images.to(device), labels.to(device)
    
    optimizer.zero_grad()
    
    output = model.forward(images)
    prob = F.softmax(output, dim=1)
    #loss = torch.nn.functional.nll_loss(torch.log(p), y)
    loss = criterion(output, labels)
    loss.backward()
    optimizer.step()
    
    running_loss += loss.item()
    
  #if steps % print_every == 0:
  valid_loss = 0
  accuracy = 0
  model.eval()
  for images, labels in dataloader_test:
    optimizer.zero_grad()
    with torch.no_grad():
       
      images, labels = images.to(device), labels.to(device)

      output = model.forward(images)
      loss = criterion(output, labels)
      p = torch.nn.functional.softmax(output, dim=1)
          
      valid_loss += loss.item()
          
      ps = torch.exp(output)
         
      top_p, top_class = ps.topk(1, dim = 1)
      equals = top_class == labels.view(*top_class.shape)
      accuracy += torch.mean(equals.type(torch.FloatTensor))
        
  print("Epoch: {}/{} " .format(epoch+1, epochs))
  print("Train loss: {:.4f}.. " .format(running_loss/len(dataloader_train)))
  print("Valid loss: {:.4f}.. " .format(valid_loss/len(dataloader_test)))
  print("Accuracy: {:.4f}.. " .format(accuracy/len(dataloader_test)))
  model.train()

Can you please tell me that how can I get the sensitivity and specifcity from this dataset? Thank you.

1 Like

Sensitivity and Specificity are usually defined for a binary classification problem.
Based on your code it looks like you are dealing with 4 classes.
In that case, you could apply a one vs. all approach, i.e. calculate the sensitivity and specificity for each class. For class0 this would be:

  • TP of class0 are all class0 samples classified asclass0.
  • TN of class0 are all non-class0 samples classified as non-class0.
  • FP of class0 are all non-class0 samples classified as class0.
  • FN of class0 are all class0 samples not classified as class0.

I think, TN means non-class0 samples classified as non-class0, right?:thinking:

2 Likes

Yeah, you are right. True Negative means all non-class elements classified as non-zero.

I got it, but how can I code it to class0 elements and non-class0 elements? to find out TP, TN, FP and FN? I am not able to find out this.

@MariosOreo Thanks for the catch! I’ve fixed it in my post. :wink:

@Deb_Prakash_Chatterj You could count it manually of create a confusion matrix first.
Based on the confusion matrix you could then calculate the stats.
Here is a small example. I tried to validate the results, but you should definitely have another look at it:

nb_samples = 20
nb_classes = 4
output = torch.randn(nb_samples, nb_classes)
pred = torch.argmax(output, 1)
target = torch.randint(0, nb_classes, (nb_samples,))

conf_matrix = torch.zeros(nb_classes, nb_classes)
for t, p in zip(target, pred):
    conf_matrix[t, p] += 1

print('Confusion matrix\n', conf_matrix)

TP = conf_matrix.diag()
for c in range(nb_classes):
    idx = torch.ones(nb_classes).byte()
    idx[c] = 0
    # all non-class samples classified as non-class
    TN = conf_matrix[idx.nonzero()[:, None], idx.nonzero()].sum() #conf_matrix[idx[:, None], idx].sum() - conf_matrix[idx, c].sum()
    # all non-class samples classified as class
    FP = conf_matrix[idx, c].sum()
    # all class samples not classified as class
    FN = conf_matrix[c, idx].sum()
    
    print('Class {}\nTP {}, TN {}, FP {}, FN {}'.format(
        c, TP[c], TN, FP, FN))
6 Likes

Yeah, it worked, but can you please explain this, like what is nb_samples and this -

conf_matrix = torch.zeros(nb_classes, nb_classes)
for t, p in zip(target, pred):
    conf_matrix[t, p] += 1

print('Confusion matrix\n', conf_matrix)

TP = conf_matrix.diag()
for c in range(nb_classes):
    idx = torch.ones(nb_classes).byte()
    idx[c] = 0
    # all non-class samples classified as non-class
    TN = conf_matrix[idx.nonzero()[:, None], idx.nonzero()].sum() #conf_matrix[idx[:, None], idx].sum() - conf_matrix[idx, c].sum()
    # all non-class samples classified as class
    FP = conf_matrix[idx, c].sum()
    # all class samples not classified as class
    FN = conf_matrix[c, idx].sum()

nb_samples defines the number of samples we are dealing with.
In your code it would be the batch size, if you apply this code in your training loop, or the length of your dataset, if you collect all predictions and targets during training.

The code first creates a confusion matrix and uses it to compute the TP, TN, FP and FN stats.
Have a look at the Wikipedia info about Confusion Matrix for more information.

Okay, I got it, should I apply after the train and test loop or in them?

I would apply it afterwards, since you would then have the statistics of the whole datasets.

1 Like

you can also check :

https://scikit-learn.org/dev/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html

1 Like