Set forget gate bias of LSTM

4

Suppose you have an LSTM model with two layers:

l = LSTM(10, 20, 2)

Then all its parameters names are in this list of lists:

>>> l._all_weights
Out[35]: 
[['weight_ih_l0', 'weight_hh_l0', 'bias_ih_l0', 'bias_hh_l0'],
 ['weight_ih_l1', 'weight_hh_l1', 'bias_ih_l1', 'bias_hh_l1']]

And you can acces them by name:

l.bias_ih_l0.data[20:40].fill_(0)

(In this case biases are arrays of length 80.)

If you want to set the bias for all forget gates to 1:

for names in l._all_weights:
    for name in filter(lambda n: "bias" in n,  names):
        bias = getattr(l, name)
        n = bias.size(0)
        start, end = n//4, n//2
        bias.data[start:end].fill_(1.)
12 Likes