LSTM/GRU gate weights

aron-bordin · May 9, 2017, 10:08pm

Hi

I would like to have a custom weight initialization to each gate of my rnn (GRU and LSTM). How can I get the weights of a specific gate in the GRU/LSTM implementation ?

ruotianluo · May 10, 2017, 1:27pm

#LSTM
net = nn.LSTM(100, 100) #Assume only one layer
w_ii, w_if, w_ic, w_io = net.weight_ih_l0.chunk(4, 0)
w_hi, w_hf, w_hc, w_ho = net.weight_hh_l0.chunk(4, 0)

#GRU
net = nn.GRU(100,100)
w_ir, w_ii, w_in = net.weight_ih_l0.chunk(3, 0)
w_hr, w_hi, w_hn = net.weight_ih_l0.chunk(3, 0)

reference:

github.com

pytorch/pytorch/blob/master/torch/nn/modules/rnn.py

import math
import torch
import warnings

from .module import Module
from ..parameter import Parameter
from ..utils.rnn import PackedSequence


class RNNBase(Module):

    def __init__(self, mode, input_size, hidden_size,
                 num_layers=1, bias=True, batch_first=False,
                 dropout=0, bidirectional=False):
        super(RNNBase, self).__init__()
        self.mode = mode
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.bias = bias

This file has been truncated. show original

Dheeraj_M_Pai · July 3, 2018, 4:40pm

How do I get the biases?

purvanshi_mehta · July 19, 2018, 4:18pm

You can get weights of all layers by

rnn = nn.LSTM(100, 100,20)
for name in rnn.named_parameters():
    if 'weight' in name[0]:
        weight_list.append(name[1])

Kris · October 3, 2018, 12:23pm

Great answer, thanks!
But I think that for the GRU, the second weight_ih_l0 should be changed to weight_hh_l0

miranda · March 18, 2020, 5:50pm

@ Dheeraj_M_Pai: You can get biases by: net.bias_hh_l0, and net.bias_ih_l0. 0 in l0 stands for layer number. If you have more than one lstm layer then change 0 to 1, 2, or the number of whichever layer you want to manipulate.