That say, I have two different losses in model.
My model is something like a backbone and followed by two branches (A and B branch )with two different losses.
My target is to only fine-tune branch A with a new dataset which only have A Label (I random B’s Label for training).
What I did is turn the backbone.requires_grad and branch B.requires_grad to False to prevent gradient and update the weights.
While training, Im still calculating both A and B loss. From my knowledge, backbone and branch B won’t update it weights since the requires_grad is turned to False; however, the branch b losses is converging which is kinda strange(As mentioned above, I random B’s Label for training). Since the B’s Label actually not the real ground-truth, the only possibility for B loss to converge is that the branch B is updating the weights.
However, I plot the B.weight and B.gradient, the weights aren’t updating and the gradient is None.
Why’s the loss still converging? Any Ideas