maybe you are trying to call a non
Anyway, if your
exp_activation is outside of the class definition, you can use it normally.
def exp_activation_square(x): return torch.exp(x) ** 2 x = Variable(torch.rand(3), requires_grad=True) y = Variable(torch.rand(3), requires_grad=True) z = exp_activation_square(x * y).sum() z.backward()
Ah, @Tudor_Berariu is right, I didn’t see all your code so I didn’t get where the problem was.
Yes, if you want to try another non-linearity which is not currently supported in
nn.RNN, you need to write your own for loop.
Although it wouldn’t be hard to add support for it in
nn, but I’m not sure it’s a very common use-case
thank you ！everyone,I will try it,
for video tutorial on activation functions
Hello I have a question for implementing activation function. How can we implement our own activation function that need parameter?, Now I want to make like thresholding function where the threshold is determined in training this is similar with PReLU but in here I have a custom additional operation.
PReLU you could implement your own activation function writing a custom
nn.Module (just like writing your model).
you can write a customized act function like below (e.g. weighted Tanh)
class weightedTanh(nn.Module): def __init__(self, weights = 1): super().__init__() self.weights = weights def forward(self, input): ex = torch.exp(2*self.weights*input) return (ex-1)/(ex+1)
I write the p-Sigmigd to have 3 learnable parameters as (Alpha, Beta, and Sigma). Would you please tell me that this implementation is correct?
class SigmoiidLearn(nn.Module): def __init__(self): super(SigmoiidLearn, self).__init__() self.Alpha = nn.Parameter(torch.ones(1), requires_grad=True) self.Beta = nn.Parameter(torch.zeros(1),requires_grad=True) self.Sigma = nn.Parameter(torch.ones(1),requires_grad=True) def forward(self,input): Sigma=self.Sigma Beta=self.Beta Alpha=self.Alpha Out=torch.zeros(input.shape) for ii in range(input.shape): for ii1 in range(input.shape): for ii2 in range(input.shape): for ii3 in range(input.shape): vv=input[ii,ii1,ii2,ii3] Out[ii,ii1,ii2,ii3]=Sigma*(1/(1+torch.exp((-1*(vv*Alpha))+Beta))) return Out
and the function that I call it as :
class Net(nn.Module): def __init__(self,ngpu,nz,ngf): super(Net, self).__init__() self.ngpu=ngpu self.nz=nz self.ngf=ngf self.l1= nn.Sequential( nn.ConvTranspose2d(self.nz, self.ngf * 8, 3, 1, 0, bias=False), nn.BatchNorm2d(self.ngf * 8), nn.ReLU(True)) self.l2=nn.Sequential(SigmoiidLearn()) def forward(self, input1): x = self.l1(input1) x = self.l2(x) return x
I’m not sure, if the nested loop is needed, since the parameters only contain one element and could maybe be applied directly yo the
However, I would recommend to use some defined input data, calculate the backward pass, and check the gradient for the expected values.
Many thanks for your reply.
Would you please tell me how I can write it without loop? to apply directly on the input?
Sorry I need to customize the Tanh function in the way that it saturates in the specific number.
I did not find any thing for Tanh. I found for sigmoid with learnable parameters.
tanh is just a rescaled version of the logistic sigmoid function as described here by @rasbt.
You could reuse the sigmoid function with learnable parameters, which you have already found, or adapt it to fit your use case.
I am very two minded and need your idea. I am using nn.LeakyReLU(0.2, inplace=True)), for my discriminator with CNN layers.
I’m not sure should I make the inplace False or keep it True in my GAN. I read some comments but were not clear.
If no error is raised, you could set it to
inplace=True and could save a bit of memory.
On the other hand, if an error is raised or you are planning to script the model, you could leave it as
but it has no effect on the final result? be False or True?
That is correct. It won’t change any results and is only a potential optimization, which could save memory.
can you tell me why it is okay not to write about backpropagation?