class Mymodel(nn.Module):
def init(self:
super(Mymodel, self).init()
self.conv1 = nn.Conv2d(...)
self.conv2= Myconv(...)
def forward(self, x, conv2=False):
if conv2:
self.conv2.weight.data = self.conv1.weight.data
out = self.conv2(x)
else:
out = self.conv1(x)
return out
model = Mymodel()
output = model(input, conv2=True)
In the above example, I want to do a forward pass in which my input is fed to conv2 instance and in backward pass, the weghts for conv1 needs to be updated. But when I do a loss.backward(), it is actually going to the backward function for conv2 instead I want it to go to backward function for conv1.
Is it possible that when I do loss.backward() it goes through the backward function for conv1?
So when I do a loss.backward() it goes to line “ctx.saved_tensors” in my backward function, but I keep on getting this error “*** RuntimeError: No grad accumulator for a saved leaf!”
What is the reason for this error?
The require grad for input and weight is True.
Also i am using double backward. Is there any problem with double backward?
What is the other way other than using *.data to make weights for self.conv2 to be equal to weights of self.conv1?
How do you get this error? Do you still use .data?
Do you always have a bias? Does it require gradients? Do you return gradient in the backward for it all the time?
What is the other way other than using *.data to make weights for self.conv2 to be equal to weights of self.conv1?
Is there a reason why they are two different Tensors? If you just want them to have the same weight Tensor, just do that in the init by creating MyConv with self.conv1.weight or by passing this to your custom forward directly.
“No grad accumulator for a saved leaf!”
I solved this error. I made requires_gard=False for the parameters for which I didn’t want to calculate the grads
“What is the other way other than using *.data to make weights for self.conv2 to be equal to weights of self.conv1?”
I have two tensors because I train with nn.conv2d and I infer with my conv implementation. What is the most efficient way for conv2 and conv1 to share weights for all the epochs. Basically the change in weight for conv1 should reflect in conv2 weight as well. I tried using the below line in my init:
But I think this would only make weight equal when init is called while initialising the model. The changes in conv1.weight won’t reflect on conv2 weights in subsequent epochs.
One way is to make the weight of conv2 and conv1 equal in forward function. But is there some other way to do so?
As I mentioned above, you just want a single Parameter for both to make sure they remain the same.
Doing self.conv2.weight = self.conv1.weight will do that.
Hey @albanD
Will the below line allocate new memory for the weights for self.conv2 or it will just use the weights from the memory where self.conv1 weight is saved.
" self.conv2.weight = self.conv1.weight"
@albanD would both the options below make conv1 and conv2 share same weights?
What I observed is that with option2 when I am updating conv1 it’s not reflected in conv2?
Not sure what is the reason for that
Option1:
class Mymodel1(nn.Module):
def init (self:
super(Mymodel, self). init ()
In option2 when I did mymodel.conv2.weight.data = mymodel.conv1.weight that actually helped to do what I was able to do with option1. But still I am curious why wouldn’t mymodel.conv2.weight =mymodel.conv1.weight won’t work in option2.