How does Autograd deal with non-differentiable opponents such as abs() and max()?

Functions like abs() are non-differentiable at some certain points. How does Autograd deal with that?

1 Like

Mostly some (more or less) arbitrary extension from the intervals is used.
One thing that people seem to like - and PyTorch mostly does - is to have zero derivative if it is zero in a neighbourhood - eg for relu at zero.

Best regards

Thomas

2 Likes

OIC. Thank you for helping!