Plot training and validation accuracy and loss of Mask RCNN for instance segmentation

vision
1

I am using Mask RCNN tutorial at

https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

I want to plot the training and validation accuracy and loss. How can I plot it.

Second problem is that after fine tuning I get a lot of masks. The only valid masks are the top The number of persons in image. All other masks are errors. How can I identify the total valid masks.

2

You could store the loss and accuracy in a list as scalar values via e.g.:

losses.append(loss.detach().cpu().item())

and use matplotlib.pyplot.plot to visualize the loss and accuracy curves.

You could probably just index the first masks given the number of people. I’m not sure I understand the problem correctly, so could you explain the issue a bit more?

3

Adding to what @ptrblck said … u can set a threshold value on scores predicted by the model and subsequently filter the corresponding masks … e-g

    outputs = model(image)
    outputs = [{k: v.to(cpu_device) for k, v in t.items()} for t in outputs]
    pred_score = list(outputs[0]['scores'].numpy())
    pred_t = [pred_score.index(x) for x in pred_score if x>=threshold][-1]
    masks = (outputs[0]['masks']>0.5).squeeze().numpy()
    masks = masks[:pred_t+1]
4

i am a little confused …

its okay with the loss, but where is “accuracy” during training in pytorch Mask R-CNN model ??

  # For Training
  images,targets = next(iter(data_loader))
  images = list(image for image in images)
  targets = [{k: v for k, v in t.items()} for t in targets]
  output = model(images,targets)

 #Output 
 {'loss_box_reg': tensor(0.0100, grad_fn=<DivBackward0>),
 'loss_classifier': tensor(0.2265, grad_fn=<NllLossBackward>),
 'loss_objectness': tensor(0.0190, grad_fn=<BinaryCrossEntropyWithLogitsBackward>),
'loss_rpn_box_reg': tensor(0.0068, grad_fn=<DivBackward0>)}

its just a dictionary of losses … they can be plotted but i am more concerned in accuracy plotting during training … any thoughts ?

5

Depending which accuracy you want you could calculate it using the model output.
How are the targets defined in your use case and which accuracy are you looking for?

6

Dont we need to have predictions from the model output in order to calculate an accuracy ?? what i was trying to say earlier (and couldnt make it clear) was that for pytorch’s Mask RCNN implementation … we need to have model in eval model in order to generate predictions whcih can be then subsequently used for accuracy calculations … the same cannot be done in model train mode … as said in its documentation here

   The model returns a Dict[Tensor] during training, containing the classification and regression
   losses for both the RPN and the R-CNN, and the mask loss.

and this is realized via eager_outputs methods in generalizedRCNN class as seen here.
To my understanding , in order to get predictions (and subsequent acc) in train mode , the only options seems to be is modification of this eager_outputs method. i have tried it but in vain (as of now).

To answer your question , i am looking for mIoU … which i am already calculting via predictions in eval mode. but since now i want to analyse model overfitting , i need to have a train_mIoU generated from the training dataset. ofcourse i can just change the dataset to training dataset in eval mode to generate train_mIoU but that adds another overhead in computations … i was looking if there is any option to get the same during model training and NOT during evaluations …

P.S thinking out of the box , any other option to analyse model overfitting ? i feel reluctant to go to train/test loss monitoring for this purpose … but open to suggestions .

7

Hi guys can you please help me to plot the training an d validationn and loss accuracy using example code along with its output. Sorry guys i didnt get the concept of all ur suggestions as i am a beginner to use mask rcnn model for my project.

thanks in advance!!!