So I am trying to use pytorch for a non-neural net application for taking local pose estimation to global pose estimation. I am trying to use a custom loss function to compute the difference between a global 3d xyz coordinates and 2d uv image pixel coordinates. My question is do gradients persist when you use
retain_grad but pass the tensor into a new tensor initialization?
I’m using a kinematic hand model for pose estimation so I am trying to find the gradients of the yaw/pitch/roll of the rotation matrix of the model wrist, translation matrix of the wrist, and flexion/abduction angles of the fingers after forward kinematics on the hand model. I randomly initialized the R, p, theta values and am using ground truth values of the image coordinates to perform gradient descent
yaw_w = torch.tensor().float() pitch_w = torch.tensor().float() roll_w = torch.tensor().float() p_wrist = torch.tensor([0., 50., 350.]) theta = torch.tensor([[0., 0., 0., 0.], [0., 0., 0., 0.], [0., 0., 0., 0.], [0., 0., 0., 0.], [0., 0., 0., 0.]])
def R_from_angs(yaw, pitch, roll): Rx = torch.tensor([[1, 0, 0], [0, torch.cos(roll), -torch.sin(roll)], [0, torch.sin(roll), torch.cos(roll)]]).float() Ry = torch.tensor([[torch.cos(pitch), 0, torch.sin(pitch)], [0, 1, 0], [-torch.sin(pitch), 0, torch.cos(pitch)]]).float() Rz = torch.tensor([[torch.cos(yaw), -torch.sin(yaw), 0], [torch.sin(yaw), torch.cos(yaw), 0], [0, 0, 1]]).float() R = torch.mm(torch.mm(Rz, Ry), Rx) return R
def loss_2d(xyz, uv_pred, c_scores): # projecting into image plane using camera intrinsics uv_proj = perspective_proj(xyz, torch.as_tensor(ci).float()) uv_diff = uv_proj - torch.as_tensor(uv_pred).float() uv_norm = torch.norm(uv_diff, p=2, dim=1).view(21,1) ** 2 loss = torch.sum(c_scores*uv_norm) return loss
yaw = torch.tensor(yaw_w, requires_grad=True) pitch = torch.tensor(pitch_w, requires_grad=True) roll = torch.tensor(roll_w, requires_grad=True) p = torch.tensor(p_wrist, requires_grad=True) theta = torch.tensor(theta, requires_grad=True) R_wrist = R_from_angs(yaw, pitch, roll) uv_pred, conf_score, xyz_pred, viz_crop = caffe_predictions(img) # forward kinematics from hand Rotation, translation, and finger theta angles xyz_fk_init = hand_fk(R_wrist, p, torch.from_numpy(avg_bone_length).float(), parent_unit_vec,theta) loss = loss2d(xyz_fk_init, uv_pred, torch.as_tensor(conf_score).float())
When I perform backpropagation on the custom loss function above I lose the gradients for yaw/pitch/roll. I think it has to do with the fact that I am creating new tensors to create the transformation matrix. Any idea?
yaw.retain_grad() pitch.retain_grad() roll.retain_grad() p.retain_grad() theta.retain_grad() loss.backward() print(p.grad) #does return a value print(pitch.grad) #gives me nothing
I know this is quite a wall of text so I really appreciate you going through and reading it!