# Precision problem in gradient check of custom function

Hi, I implemented my custom function and use the gradcheck tool in pytorch to check whether there are implementation issues. While it did not pass the gradient checking because of some loss of precision.

I set eps=1e-6, atol=1e-4. But I did not find the issue of my implementation.
Suggestions would be appreciated.

Edit: I post my code scripts here:

@staticmethod
def forward(ctx, input, weight, bias=None):
ctx.save_for_backward(input, weight, bias)
ctx.num = input.size(0)
output = input.clone().zero_()
for i in range(ctx.num):
output[i,:] = torch.mul(input[i,:], weight)

if bias is not None:
output += bias

return output
@staticmethod
input, weight, bias = ctx.saved_variables

for i in range(ctx.num):

for i in range(ctx.num):

if bias is not None and ctx.needs_input_grad[2]:

if bias is not None:
else:

What order of magnitude are the gradients? A general rule of thumb is to set eps to 1e-6 times the maximum magnitude of the gradient because floating-point precision is usually accurate to 6 decimal places.

The goal of my custom function is to calculate Hadamard(pointwise) product of input and weights, while the weights are parameters to be trained.
The scripts above can also be simplified as follows, I did not figure out my implementation errors.

@staticmethod
def forward(ctx, input, weight, bias=None):
ctx.save_for_backward(input, weight, bias)

output = torch.mul(input, weight)
if bias is not None:
output += bias

return output
@staticmethod
input, weight, bias = ctx.saved_variables

if bias is not None and ctx.needs_input_grad[2]:

if bias is not None:
else:

@richard
I have figured out an implementation error: