Why does the .clamp function exist?

1

I was looking at the example:

http://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_autograd.html

and has the line:

# Forward pass: compute predicted y using operations on Variables; these
# are exactly the same operations we used to compute the forward pass using
# Tensors, but we do not need to keep references to intermediate values since
# we are not implementing the backward pass by hand.
y_pred = x.mm(w1).clamp(min=0).mm(w2)

I read the documentation for clamp:

torch.clamp(input, min, max, out=None) ā†’ Tensor
Clamp all elements in input into the range [min, max] and return a resulting Tensor.

http://pytorch.org/docs/master/torch.html#torch.clamp

but it didnā€™t make sense to me. Why do we need to do such a weird thing? Tensorflow doesnā€™t ā€œclampā€ anything during matrix multiplication why does pytorch?

4 Likes
2

clamp(min=0) is exactly ReLU.

20 Likes
3

Oh I see. Thanks. So there is no relu function? we just use clamp?

I was very confused cuz I was just trying to implement linear regression and there was a random clamp command in most examples Iā€™ve seen which was confusing to me.

4

Of course there is. Relu in PyTorch is torch.nn.functional.relu. So

import torch.nn.functional as F
y_pred = F.relu(x.mm(w1)).mm(w2)

should work in same way.

1 Like
5

niceā€¦ :+1:

6

Someone would use clamp to clip the rewards in reinforcement learning.

5 Likes
7

Or to clip the weight values in a Wasserstein GAN

4 Likes
8

There are relu, and relu6. However what if you want some thing like relu7?

1 Like
9

I think I have a pretty good grasp of the situations which might lead one to use ReLU6; but Iā€™m kinda interested in itā€™s history. Got any good links?

1 Like
10

Thatā€™s exactly the same as numpy.clip.

2 Likes
11

import torch.nn.functional as F
y_pred = F.threshold(x.mm(w1), 0, 7).mm(w2)

is it right?

12

I donā€™t think F.threshold can accept min and max.

x.mm(w1).clamp(0,7).mm(w2)

1 Like
13

import torch.nn.functional as F
y_pred = F.hardtanh(x.mm(w1), 0, 7).mm(w2)

I found my mistake. Not threshold.

Just use hardtanh!!

14

In detection, i just found the code use clamp to ensure right poit minus lift point more than 0.

15

Could also use clamp() to constrain the range of your update tensors in SGD optimization