nn.RNN supports only tanh or relu for nonlinearity [code]. One easy way to solve your problem is to write your own loop over the sequence just like in this example.
2 Likes
maybe you are trying to call a non @staticmethod?
Anyway, if your exp_activation is outside of the class definition, you can use it normally.
def exp_activation_square(x):
return torch.exp(x) ** 2
x = Variable(torch.rand(3), requires_grad=True)
y = Variable(torch.rand(3), requires_grad=True)
z = exp_activation_square(x * y).sum()
z.backward()
1 Like
Ah, @Tudor_Berariu is right, I didn’t see all your code so I didn’t get where the problem was.
Yes, if you want to try another non-linearity which is not currently supported in nn.RNN, you need to write your own for loop.
Although it wouldn’t be hard to add support for it in nn, but I’m not sure it’s a very common use-case
1 Like
thank you ！everyone,I will try it,
visit https://quickkt.com/tutorials/artificial-intelligence/deep-learning/activation-function/
for video tutorial on activation functions
Hello I have a question for implementing activation function. How can we implement our own activation function that need parameter?, Now I want to make like thresholding function where the threshold is determined in training this is similar with PReLU but in here I have a custom additional operation.
1 Like
Similar to PReLU you could implement your own activation function writing a custom nn.Module (just like writing your model).
1 Like
you can write a customized act function like below (e.g. weighted Tanh)
class weightedTanh(nn.Module):
def __init__(self, weights = 1):
super().__init__()
self.weights = weights
def forward(self, input):
ex = torch.exp(2*self.weights*input)
return (ex-1)/(ex+1)Hi @ptrblck thank you for your reply, could you give a simple example?, I have tried several things like this discussion , but its seem failed.
Hi Ptrblck,
I write the p-Sigmigd to have 3 learnable parameters as (Alpha, Beta, and Sigma). Would you please tell me that this implementation is correct?
class SigmoiidLearn(nn.Module):
def __init__(self):
super(SigmoiidLearn, self).__init__()
self.Alpha = nn.Parameter(torch.ones(1), requires_grad=True)
self.Beta = nn.Parameter(torch.zeros(1),requires_grad=True)
self.Sigma = nn.Parameter(torch.ones(1),requires_grad=True)
def forward(self,input):
Sigma=self.Sigma
Beta=self.Beta
Alpha=self.Alpha
Out=torch.zeros(input.shape)
for ii in range(input.shape[0]):
for ii1 in range(input.shape[1]):
for ii2 in range(input.shape[2]):
for ii3 in range(input.shape[3]):
vv=input[ii,ii1,ii2,ii3]
Out[ii,ii1,ii2,ii3]=Sigma*(1/(1+torch.exp((-1*(vv*Alpha))+Beta)))
return Out
and the function that I call it as :
class Net(nn.Module):
def __init__(self,ngpu,nz,ngf):
super(Net, self).__init__()
self.ngpu=ngpu
self.nz=nz
self.ngf=ngf
self.l1= nn.Sequential( nn.ConvTranspose2d(self.nz, self.ngf * 8, 3, 1, 0, bias=False),
nn.BatchNorm2d(self.ngf * 8),
nn.ReLU(True))
self.l2=nn.Sequential(SigmoiidLearn())
def forward(self, input1):
x = self.l1(input1)
x = self.l2(x)
return x
I’m not sure, if the nested loop is needed, since the parameters only contain one element and could maybe be applied directly yo the input tensor.
However, I would recommend to use some defined input data, calculate the backward pass, and check the gradient for the expected values.
Many thanks for your reply.
Would you please tell me how I can write it without loop? to apply directly on the input?
HI Ptrblck,
Sorry I need to customize the Tanh function in the way that it saturates in the specific number.
I did not find any thing for Tanh. I found for sigmoid with learnable parameters.
tanh is just a rescaled version of the logistic sigmoid function as described here by @rasbt.
You could reuse the sigmoid function with learnable parameters, which you have already found, or adapt it to fit your use case.
1 Like
HI Ptrblck,
I am very two minded and need your idea. I am using nn.LeakyReLU(0.2, inplace=True)), for my discriminator with CNN layers.
I’m not sure should I make the inplace False or keep it True in my GAN. I read some comments but were not clear.
Many thanks
If no error is raised, you could set it to inplace=True and could save a bit of memory.
On the other hand, if an error is raised or you are planning to script the model, you could leave it as inplace=False.
1 Like
but it has no effect on the final result? be False or True?
That is correct. It won’t change any results and is only a potential optimization, which could save memory.
1 Like