System freezes on a GRU forward pass

Hello. I am having trouble over a small piece of code.

import torch
import torch.nn as nn
import math

class Segmentation(torch.nn.Module):
    def __init__(self, input, hidden, output, layers, bidirectional, kernal, dropout_prob):
        assert(kernal[0]%2==1 and kernal[1]%2==1), 'Convolution kernal sizes should be odd for keeping same dimensions.'
        padding = (math.floor(kernal[0]/2), math.floor(kernal[1]/2))

        # cnn input = [N * C * H * W]   H=time, W=Frequency

        self.conv_1 = nn.Conv2d(1, 1, kernel_size=kernal, padding=padding)
        self.mp_1 = nn.MaxPool2d(kernel_size=kernal, stride=1, padding=padding)
        self.conv_2 = nn.Conv2d(1, 1, kernel_size=kernal, padding=padding)
        self.mp_2 = nn.MaxPool2d(kernel_size=kernal, stride=1, padding=padding)

        self.dropout = nn.Dropout(p=dropout_prob)

        self.gru = nn.GRU(input, hidden, num_layers=layers, 
                    dropout=dropout_prob, bidirectional=bidirectional, batch_first=True)
        in_features = hidden*2 if bidirectional else hidden
        self.linear = nn.Linear(in_features=in_features, out_features=output)
        self.log_softmax = nn.LogSoftmax(dim=-1)

    def forward(self, input, input_lens):
        # inputs must be length wise sorted
        # input is already padded
        input = self.mp_1(self.conv_1(input))
        input = self.mp_2(self.conv_2(input))
        input = self.dropout(input)
        # now pack the sequences.
        # input -> [batch, ch=1, height=time, width=frequency]
        input = input.squeeze(1)
        packed_input = nn.utils.rnn.pack_padded_sequence(input, input_lens, batch_first=True)
        packed_output, _ = self.gru(packed_input)
        output, _ = nn.utils.rnn.pad_packed_sequence(packed_output, batch_first=True)

        output = self.linear(output)
        return output

The problem occurs on line 39 packed_output, _ = self.gru(packed_input). After the tesnor is passed to the gru layer, the system just freezes. I didn’t understand what was happening at first. Then I opened the debugger and system monitor and found that after coming to this line, the RAM consumption starts increasing. My RAM is 8GB and the programme alone consumes more than 7 GB, which is really weird, and therefore the system crushes. Every time I run this script, I had to force restart my machine. I tried googling the issue, NO LUCK!
the input to the self.gru(packed_input is a packed sequence.
The only reason I can think of this kind of behavior is an infinite for loop. But How can that happen on the forward layer of a built in library?
Or can it?


The fact that you are using 7GB is not necessarily weird. Can you try decreasing the batch size, and see if you still have the same problem? Also if you train this model without GPU it will take you forever.

You are right. Memory seems to be not enough. I am actually working with speech audio data (MFCC features) so the input to the gru has a shape of [128 * 1500(aprx) * 20], 128:-batch size, 1500- sequence length in time, 20:- feature vector size. the corresponding audio files are anywhere from 2-10 seconds. This shape is actually huge for the GRU. I tried batch size of 1, and it still crashes. Any idea about a solution?


In fact you are applying GRU to the output of the second convolution if I am not wrong. And I think this is maybe a heavy operation (depending on the number of kernels in you convolution).

If you can’t fit this load in memory, the only possibilities for you are to have better hardware (you can use colab, it may provide you with a GPU of 15 GB), or you can change the architecture, or reduce the input size, (if you change the parameters that you use to compute MFCC features, you can get denser features)

1 Like

I understand what you are saying. Thank you for your reply!