Low validation accuracy and loss for multi-output classifier

Hi,

I’m trying to build multi-output image classifier based on MobileNetV2.
First output has 2 classes and secondary has 7. I have balanced train and validate data (each image is labeled by one of value from both outputs):

  • validate: 424 for each of 7 classes and 1484 for each of 2 classes
  • train: 1708 for each of 7 classes and 5978 for each of 2 classes
    I have two separate nn.Sequentail outputs, for loss I’m using cross_entropy and Adam as optimizer.
    During training, accuracy of each of outputs is increasing and total loss decreasing but every 25 epochs I’m validating data using saved checkpoints and unfortunately I got a problem with increasing loss and low accuracy score. For validation it looks like that:
  • After 25 epochs: loss - 2.31, first_class_acc - 0.79, second_class_acc - 0.19
  • After 50 epochs: loss - 2.13, first_class_acc - 0.85, second_class_acc - 0.25
  • After 200 epochs: loss - 3.30, first_class_acc - 0.88, second_class_acc - 0.24
  • After 300 epochs: loss - 4.54, first_class_acc - 0.88, second_class_acc - 0.24
    Do you have an idea what probably am I doing wrong?