Don't do autodiff when ReLU it's not differentiable

Hi everyone,

Do you know if it’s possible to train a model with the condition to not do the back propagation when it’s not differentiable ? For exemple, I want to throw away the input if at some point I do ReLU’(0).

class MyReLU(torch.autograd.Function):

@staticmethod
def forward(ctx, input):
    if input == 0:
        ctx.mark_non_differentiable(input)
        return input.clamp(min=0)
    else: 
        ctx.save_for_backward(input)
        return input.clamp(min=0)

@staticmethod
def backward(ctx, grad_output):
    input, = ctx.saved_tensors 
    grad_input = grad_output.clone()
    grad_input[input < 0] = 0
    return grad_input  

Do you think it’s possible to do it ?

Thanks