I am trying to rebuild a Keras architecture in pytorch, which looks like this
rnn_layer1 = GRU(25) (emb_seq_title_description) # [...] main_l = Dropout(0.1)(Dense(512,activation='relu') (main_l)) main_l = Dropout(0.1)(Dense(64,activation='relu') (main_l)) #output output = Dense(1,activation="sigmoid") (main_l)
So I tried to adjust the basic RNN example in pytorch and add ReLUs to the Linear layers. However, I am not sure if I can call ReLU directly in the forward method or should call it in the init method.
My first try looks like this:
import torch.nn as nn class RNN(nn.Module): def __init__(self, input_size, hidden_size, output_size): super(RNN, self).__init__() self.hidden_size = hidden_size self.i2h = nn.Linear(input_size + hidden_size, hidden_size) self.i2o = nn.Linear(input_size + hidden_size, output_size) def forward(self, input, hidden): combined = torch.cat((input, hidden), 1) hidden = nn.ReLU(self.i2h(combined)) output = nn.ReLU(self.i2o(combined)) output = nn.sigmoid(output) return output, hidden def initHidden(self): return torch.zeros(1, self.hidden_size) n_hidden = 128 n_letters = 26 n_categories = 2 rnn = RNN(n_letters, n_hidden, n_categories)
However, when I look at the rnn object in python, I will not see the ReLUs, so maybe it’s not right to call nn.ReLU directly in the forward method…
RNN( (i2h): Linear(in_features=154, out_features=128, bias=True) (i2o): Linear(in_features=154, out_features=2, bias=True) )