Hello, basically I’m trying to implement ReLU from scratch without using numpy. In numpy the simple implementation is np.maximum(input,x) and the torch equivalent of maximum is max. But using torch.max as torch.max(input, 0) basically computes the maximum value along the 0th axis.

One way to do is to create an all-zero matrix and then use torch.max but it’s inefficient. Any other things I can use?

Thanks, clamp works for ReLU but not for prime ReLU, output = input.clamp(max=1) doesn’t work since most entries are between 0 and 1.
torch.where works fine but all these are still slow compared to output=numpy.maximum(input,0) by a wide margin.
I tried searching the official documentation for ReLU but the actual implementation seems to be in lua which I couldn’t understand. Also, does pytorch has an implementation for ReLU prime or is it computed internally in the autograd