Thank you both for your answers.
My code is quite spread out in numerous files, so I could not include it; also it was hard to do a minimal reproducible example.
I’m using version 0.2.0.
Thank you for suggesting this. It helped me track down the following operation. Here, I just want to modify a parameter inside a module:
self.my_param.data = other_tensor # where my_param is a nn.Parameter
# other_tensor has a different size than self.my_param.data
Then I do a forward and a backward pass, without any problem. Then, I modify the parameter again. I do a forward pass, then a backward pass, and there I have the RuntimeError. The reason is that
.grad is allocated on the first backward pass. The second time I change
.data, the size of
.data becomes different from the size of
Thus the error:
RuntimeError: invalid argument 3: sizes do not match
My solution to this was to replace the parameter completely instead of assigning to
self.my_param = nn.Parameter(other_tensor)
Now everything works.
Should this be prevented?
If we want to prevent this from happening, there are many options.
- Should it be forbidden to assign a new tensor to
- Should it be forbidden to assign a tensor of a different size to
my_param.grad be reassigned automatically when the size of
- Should the sizes of all
.grad be matched to their
.data when calling
Options 2 and 4 make sense to me and seem not to complicated to implement. But in any case, it will add overhead.