Tensor of variable size without gradient

Hi
I am trying to use a tensor which stores outputs of each LSTM operation.
For example,

def __init__(self):
    self.register_buffer('h_list', torch.FloatTensor(10, 5, 20).detach())
    self.h_list = self.h_list.cuda()
    self.net = net

def reset_h_list(self):
    self.register_buffer('h_list', torch.FloatTensor(10, 5, 20).detach())
    self.h_list = self.h_list.cuda()

def forward(self, input):
    h = self.h_list[:, -5, :].contiguous().view(10, 100, -1)
    next_h = self.net(input)
    self.h_list = torch.cat((self.h_list, next_h), dim=1)

    return next_h

h_list should be initialize after end of the LSTM with the size (10, 5, 20).

Additionally, it is not trainable data at all since it just store the outputs to stack the feature of every LSTMCell iteration.

However, the memory usage is getting bigger when I run this code.
If you want additional explanation, please ask me
Thank you