This is not correct. Pytorch will not overwrite previous values. Being the same module/parameter or not is not related to how the dynamic graph is built. It will be different links in the graph. The variables saved for backward pass will be stored differently. See the following simple example:
>>> l1 = nn.Linear(3, 3)
>>> l1.weight.data.fill_(0)
>>> l1.bias.data.fill_(0)
>>> x = Variable(torch.ones(2, 3))
>>>
>>> # backward one loss only
>>> loss1 = (l1(x) - 1).abs().sum()
>>> loss1.backward()
>>> l1.weight.grad
Variable containing:
-2 -2 -2
-2 -2 -2
-2 -2 -2
[torch.FloatTensor of size 3x3]
>>>
>>> # backward the other loss only
>>> l1.weight.grad = None
>>> loss2 = (l1(x) + 1).abs().sum()
>>> loss2.backward()
>>> l1.weight.grad
Variable containing:
2 2 2
2 2 2
2 2 2
[torch.FloatTensor of size 3x3]
>>>
>>> # backward both losses together
>>> l1.weight.grad = None
>>> loss1 = (l1(x) - 1).abs().sum()
>>> loss2 = (l1(x) + 1).abs().sum()
>>> (loss1+loss2).backward()
>>> l1.weight.grad
Variable containing:
0 0 0
0 0 0
0 0 0
[torch.FloatTensor of size 3x3]