I am trying to implement NashMTL on a multi-task model. I don’t get this error initially where an MOO is not applied, where after calculating each loss, i immediately just call backward on each loss inside the task loop.
For the method, I am trying to assign each loss inside a losses tensor and i would pass this tensor to first compute the gradients for each loss respect to the shared parameters (custom shared parameter extraction function). Once i have the gradients tensor, this would be passed to the NashMTL calculation where it will derive a weighting for each of the losses. Finally, it would call a backward for the weighted and combined loss. Depending on whether I collect the loss and then apply autograd or directly apply it, I have the same error. This a complicated model and has custom loss function, I am also using DDP module since i am using multiple GPUs.
Using the anomaly detection doesn’t show anything. The Tensors involved in the backward or autograd are the losses, but I don’t understand why the tensor in the error has a shape of 256. Could anyone give me a direction to look into?
Start by working through the discussion of inplace-modification errors and
how to debug them given in this post:
If you’re still having problems, tell us what specific debugging techniques
you’ve tried and post the entire error message you get including the
forward-pass traceback produced by anomaly detection.
hello, thank you for your reply. I have read many solutions, but since my model is not that simple, It’s harder to find exactly where the error is happening. Here is the main parts of the code that i am dealing with.
I initialize grads and train_losses before the task loop inside batch loop (I only have two tasks).
Compute loss for each, compute gradients for each loss using autograd and save the losses and grads in the variables.
Outside the loop i send these grads and losses to the weighting backward which is the multi-objective optimization method NashMTL (this is from the public MTL framework LibMTL on github).
My error occurs at the final stage of NashMTL where i would perform the final backward
Setting the anomaly detection doesn’t show me anything, I’ve tried putting it on every file that might cause the error.
What i’ve done:
Replaced all the seemingly in place operations inside the Loss function file and where the model is being built (its a yolo based model)
Thought combining the losses into a tensor might be causing the problem, so tried separating each loss and adding it one by one as commented out
My question:
Does it have anything to do with the loss being collected in a tensor? Or is it happening in the loss calculation or model structure?
Also would the shared parameters have anything to do with it? (If its wrong would it cause such problem).
Maybe I am using anomaly detection wrong, how and where should i insert it? It doesnt show anything no matter where i place it.
Additionally, I found out that the tensor in question is in one of the parameter tensors during foward pass or gradient during autograd.grad backward. Before i recheck forward, and loss i want to know if any simple operations that i am performing in the previous images are causing the problem, because before applying the nashmtl, if i just calculate loss and apply .backward() for each task, I dont get any error. So i thought there is something wrong with how im handling the loss tensor afterwards, or the shared parameters.
Thanks, i found the problem. For the Anomaly detection, i tried wrapping the loss calculation, gradient calculation and just placing it at the beginning of the file, it tells me that there is an inplace operation, but did not show me exactly where or any other tracebacks.