I was transferring something written in Caffe to Pytorch and noticed a different in gradient of the MSELoss, so I tested the MSELoss of pytorch using this script:
import torch
import numpy as np
import torch.nn as nn
from torch.autograd import Variable
grads = {}
def save_grad(name):
def hook(grad):
grads[name] = grad
return hook
#pred = np.load('caffe-spn/data.npy')
#tar = np.load('caffe-spn/label.npy')
pred = np.random.rand(1,1,3,3)
tar = np.random.rand(1,1,3,3)
pred = Variable(torch.from_numpy(pred),requires_grad=True).cuda()
tar = Variable(torch.from_numpy(tar),requires_grad=False).cuda()
pred.register_hook(save_grad('pred'))
criterion = nn.MSELoss(size_average=False)
loss = criterion(pred,tar)
#loss = torch.sum((pred - tar)**2)
loss.backward()
print(loss)
print(torch.sum((pred - tar)**2))
print('prediction:')
print(pred[0,0,:,:])
print('target:')
print(tar[0,0,:,:])
print('gradient:')
print(grads['pred'][0,0,:,:])
Here’s what I got:
Variable containing:
0.7841
[torch.cuda.DoubleTensor of size 1 (GPU 0)]
Variable containing:
0.7841
[torch.cuda.DoubleTensor of size 1 (GPU 0)]
prediction:
Variable containing:
0.7884 0.7303 0.9125
0.3856 0.5309 0.5715
0.4588 0.4232 0.2187
[torch.cuda.DoubleTensor of size 3x3 (GPU 0)]
target:
Variable containing:
0.9345 0.1918 0.6964
0.1005 0.2449 0.1909
0.2013 0.5728 0.0471
[torch.cuda.DoubleTensor of size 3x3 (GPU 0)]
gradient:
Variable containing:
-0.2922 1.0770 0.4321
0.5702 0.5721 0.7612
0.5150 -0.2991 0.3432
[torch.cuda.DoubleTensor of size 3x3 (GPU 0)]
My question is that the gradient of (pred - tar)^2 w.r.t to pred should be (pred - tar), but it seems not the case in pytorch MSELoss, this could be a silly mistake, but I just wonder what I did wrong here.
