Initialize hidden layer in RNN network

1

Hello, I read similar topic in initializing hidden layer in RNN network.
However they are quite confusing for me.
Right now I have the code as follows to initialize hidden layer with zeros. Could explain to me how to modify it so that it is initialized based on training? Thx Matt

def initHidden(self):
return torch.zeros((self.MiniBatchSize,self.HiddenNodes),dtype=torch.double)

2

Do you mean you want to treat your initial hidden state as a learnable parameter? Wrap it in nn.Parameter:

class RNN(nn.Module):
    def __init__(self, ...):
        ...
        h_0 = torch.zeros((self.MiniBatchSize, self.HiddenNodes),
                          dtype=torch.double)
        self.h_0 = nn.Parameter(h_0)
        ...
3

Will I still use init_hidden between minibatches?

4

You wouldn’t want to if you are using a stateful RNN at least, where you pass in the hidden state output from the previous minibatch.

5

Thanks- I’ll try the approach!

6

How will I call the function. Previously I wrote:
hidden=model.initHidden()
outA, hidden=aModel(input, hidden)

7

Maybe I don’t call it anymore. Just use self.hidden internally and call outA=model(Input).
I still need to do this detach call I think, between minibatches I would type model.hidden_0.detach()

8

Something like this perhaps?

def forward(self, data, hidden):
    if hidden == None:
        hidden = self.h_0
    ...
9

Could you review the entire code (psueducode).

class RNN(nn.Module):
    def __init__(self, ...):
        ...
        h_0 = torch.zeros((self.MiniBatchSize, self.HiddenNodes),
                          dtype=torch.double)
        self.h_0 = nn.Parameter(h_0)
        self.Lin1 = nn.Linear(InNodes , NumNodes)
        self.Lin2= nn.Linear(InNodes , NumNodes)
   def forward(self, data, hidden):
     if hidden == None:
         hidden = self.h_0
     aDat=torch.cat((data,hidden))
     kOut=Lin1(aDat)
     hidden=Lin2(aDat)
     return kOut,hidden

aModel=RNN()
hidden=None
while True:
     aData, aTruth=Minibatch
     for count in range(numSteps):
         kOut,hidden=aModel(aData[count],hidden)
     loss=criterion(kOut,aTruth)
     hidden.detach()
10

I dont know how to format the code on this webpage.

11

I’ve formatted your code. :wink:
You can add code with three backticks before and after the code block (```).

12

I have to say people on this forum are quite delightful. :wink:

2 Likes