How to train two dataset on one model

I’ve two face image dataset, one for face alignment regression, one for age regression. If I use one model to regression those two task like this:

class Net(nn.Module):
  def forward(self, x):
        x = self.features(x)
        x1 = self.features1(x)
        x2 = self.features2(x)
        return [x1, x2]  

class LossFN(nn.Module):
   def forward(self, input, target):
    landmark_loss = self.L1Loss1(input[0][0], target[0])
    age_loss = self.L1Loss2(input[1][1], target[1])
    loss = self.alignment_ratio * landmark_loss + age_loss
    return loss

for image, label in dataloader:
  image1 = image[0]
  image2 = image[1]
  output = []
  loss = loss_fn(output, label)

When I use this process for two task(regression and classification), the loss goes well, but when I use for two regression task, training loss goes well, while the valid loss not. The wired thing is when I switch eval mode to train mode for validating, the loss goes well. Is there any possible reason for this?