MySigmoid(torch.autograd.Function)

autograd
1

I am just experimenting with defining new autograd Function for sigmoid MySigmoid similar to MyReLU in the tutorial, but the output for loss is printing nan

52 AM
52 AM1536×416 49.1 KB

What is wrong?

2

Not sure if it is root of cause, but you shouldn’t use numpy functions.

3

same problem even if I try torch.exp

4

I just tried your code on master and the forward works fine. When are you seeing loss becoming nan? First forward? After a backward?

5

Sorry to revive this topic:
I managed to implement a working version of MySigmoid, but Im also having some issues:

import torch

class MySigmoid(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        sigmoid_eval = 1.0/(1.0 + torch.exp(-input))
        return sigmoid_eval

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_temp = grad_output.clone()
        grad_input = torch.exp(-input)/((1.0 + torch.exp(-input)).pow(2))
        return grad_input

dtype = torch.float
device = torch.device("cpu")

N, D_in, D_out = 1000, 1, 1
alpha = 1e-2



x = torch.cat(
                (   torch.randn(N, D_in, device=device, dtype=dtype) ,
                    torch.ones([N,1], dtype=dtype, device=device)   ),
                1)
y = x[:,0:1]*3.5 + 2.5*torch.randn(N,1)
w = torch.randn(D_in+1, D_out, device=device, dtype=dtype, requires_grad=True)

print(w.shape)
print(w.dtype)

for t in range(5000):
    sigmoid = MySigmoid.apply
    
    y_pred = sigmoid(x.mm(w))

    loss = (y_pred - y).pow(2).sum()
    print(t, loss.item())

    loss.backward()
    
    with torch.no_grad():
        w -=  alpha * w.grad
        w.grad.zero_()

I made sure the dimensions were correct, but still having this issues:

  • If I try to multiply w by a scalar I get this:
Traceback (most recent call last):
  File "test.py", line 49, in <module>
    w -=  alpha * w.grad
TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'
  • The convergence is taking too long (reason why Im trying to modify starting w values)

I would appreciate if someone could give a lock to this idea, which I think is very useful for those of us who have just started using torch. Thanks in advance

6

change to :blush:
grad_input =grad_temp* torch.exp(-input)/((1.0 + torch.exp(-input)).pow(2))