Is torch.clamp the same as torch.nn.functional.hardtanh?

1

The following codes give the same results. Does it mean torch.clamp is totally the same as torch.nn.functional.hardtanh both in training and test phase?

x1 = torch.randn(10)
x2 = x1.detach().clone()

x1.requires_grad_()
x2.requires_grad_()

y1 = torch.nn.functional.hardtanh(x1)
y2 = torch.clamp(x2, -1.0, 1.0)

l1 = y1.sum()
l2 = y2.sum()

l1.backward()
l2.backward()

assert (x1.grad - x2.grad).abs().sum() == .0
assert l1 == l2
2

The definition of torch.Hardtanh matches the way you clamped x2. For these choices of min,max, torch.clamp seems to be identical to torch.Hardtanh.

3

Hi, what do you mean by “clamped x2”. I thought if min and max are the same, it means the behavior of clamp and hardtanh are identical, am I right?

4

I meant this line of your code:

y2 = torch.clamp(x2, -1.0, 1.0)

Yes, this is what it looks like to me.

6

~
So the implementation of hardtanh and clamp should be completely the same (both the forward and the backward) ?

7

I don’t know. You could look at the source code for both, or wait for someone knowledgeable to answer.