Based on the description of your error I think you might be hitting this issue.
I.e. you are updating (some) parameters and try to calculate the gradients (from another loss) afterwards using the already updated parameters and the now stale intermediate forward activations (which is wrong).
In this case you would either have to rerun the forward pass or calculate the gradients in another way (e.g. via backward(inputs=...)).