# Help needed - Cuda out of memory

**URL:** <https://discuss.pytorch.org/t/help-needed-cuda-out-of-memory/146915>\
**Category:** autograd\
**Created:** [March 19, 2022, 11:04pm UTC](https://discuss.pytorch.org/t/help-needed-cuda-out-of-memory/146915 "2022-03-19T23:04:53Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![mark\_eu](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ebca7d/32.png) [@mark\_eu](https://discuss.pytorch.org/u/mark_eu)\
**Post date:** [March 19, 2022, 11:04pm UTC](https://discuss.pytorch.org/t/help-needed-cuda-out-of-memory/146915/1 "2022-03-19T23:04:53Z")

</div>

Hi,

I need help. I have 32Gb of GPU and facing the problem of “Cuda out of memory.”  
I tried to improve my train routine:

```auto
import gc
def train(model, iterator, optimizer, criterion):
    
    epoch_loss = 0.0
    epoch_acc = 0.0
    
    model.train()
    
    for batch in iterator:
        
        optimizer.zero_grad()
        
        predictions = model(batch.posts) 
        
        loss = criterion(predictions, batch.type_numeric.long().to(device))
        
        acc = categorical_accuracy(predictions, batch.type_numeric.long().to(device))
        
        loss.backward()
        
        optimizer.step()
        
        epoch_loss += loss.item()
        epoch_acc += acc.item()
        #####################
        #the following part really slows the code execution
        del predictions
        del loss
        del acc
        del batch
        gc.collect()
        torch.cuda.empty_cache()
        #####################
    return epoch_loss / len(iterator), epoch_acc / len(iterator)

```

and also, I lowered the batch size to have small values:

```auto
train_batch_size = 4
valid_batch_size = 4
test_batch_size = 4

```

but still, I get the error Cuda out of memory:

![error2](https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4a31ea8449487adb299b17a6b54dd4a7be6bd2f.jpeg)

My task is an NLP multiclass classification task on a dataset that has, after preprocessing, 1 Gb. I tried to reduce MAX\_VOCAB\_SIZE to 300. I use the CNN model with the parameters:

```auto
INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 300
N_FILTERS = 100
FILTER_SIZES = [2,3,4]
OUTPUT_DIM = len(LABEL.vocab)
DROPOUT = 0.2
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]

model = CNN(INPUT_DIM, EMBEDDING_DIM, N_FILTERS, FILTER_SIZES, OUTPUT_DIM, DROPOUT, PAD_IDX)

optimizer = optim.Adam(model.parameters())

criterion1 = nn.CrossEntropyLoss()

model = model.to(device)
criterion1 = criterion1.to(device)

```

Could you please help me with this - my deadline is approaching, and I have no other ideas on how to solve this issue.

Thank you

---

<div class="post-metadata">

**Author:** ![eqy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/32/46233_2.png) [@eqy](https://discuss.pytorch.org/u/eqy)\
**Post date:** [March 20, 2022, 12:50am UTC](https://discuss.pytorch.org/t/help-needed-cuda-out-of-memory/146915/2 "2022-03-20T00:50:50Z")

</div>

It’s a bit difficult to pinpoint this without more context on the model (e.g., how many parameters, layers, etc?) and is the OOM happening on the first iteration or after many iterations due to gradually increasing memory consumption?

If it is happening in the first few iterations you might want to check if the model size itself needs to be reduced, otherwise it might be worthwhile to see which operation is gradually increasing the total memory consumed over many iterations (you can check this by removing operations until the memory used stops increasing).

---

<div class="post-metadata">

**Author:** ![mark\_eu](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ebca7d/32.png) [@mark\_eu](https://discuss.pytorch.org/u/mark_eu)\
**Post date:** [March 20, 2022, 10:10am UTC](https://discuss.pytorch.org/t/help-needed-cuda-out-of-memory/146915/3 "2022-03-20T10:10:18Z")

</div>

@eqy Thank you!  
My model has the following structure:

```auto
import torch.nn as nn
import torch.nn.functional as F

class CNN(nn.Module):
    def __init__ (self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim, 
                 dropout, pad_idx):
        
        super(). __init__ ()
        
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        
        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1, 
                                              out_channels = n_filters, 
                                              kernel_size = (fs, embedding_dim)) 
                                    for fs in filter_sizes
                                    ])
        
        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)
        
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, text):
        
        text = text.permute(1, 0)
                
        embedded = self.embedding(text)
                
        embedded = embedded.unsqueeze(1)
        
        conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]
            
        pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]
  
        cat = self.dropout(torch.cat(pooled, dim = 1)) 
        return self.fc(cat)

```

The model has 365,716 trainable parameters (in a ‘reduced’ version of MAX\_VOCAB\_SIZE = 300 instead of 25000 and batch sizes of 4, instead of 256/64/64). The OOM happens on the first iteration.

Do you have any suggestions for solving this problem? I also hope that @ptrblck and @albanD could give some advice(s). The problem does not exist with the same model on the small dataset (70MB)  
Thank you

---

<div class="post-metadata">

**Author:** ![mark\_eu](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ebca7d/32.png) [@mark\_eu](https://discuss.pytorch.org/u/mark_eu)\
**Post date:** [March 22, 2022, 11:56am UTC](https://discuss.pytorch.org/t/help-needed-cuda-out-of-memory/146915/4 "2022-03-22T11:56:01Z")

</div>

It seems to me that the problem appears in :  
`conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs`  
but I have no idea how to solve it.

Also, for text vocabulary I use:

```auto
TEXT.build_vocab(train_ds, 
                 min_freq = 3,
                 max_size = MAX_VOCAB_SIZE,
                 vectors = 'fasttext.en.300d',
                 unk_init = torch.Tensor.normal_
                 )

```

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 23, 2022, 5:41am UTC](https://discuss.pytorch.org/t/help-needed-cuda-out-of-memory/146915/5 "2022-03-23T05:41:57Z")

</div>

You are currently storing all activation outputs of `self.convs` in the `conved` list and are then applying the pooling on each of them.  
Could you combine these lines of code and see if the memory footprint would go down?

---

<div class="post-metadata">

**Author:** ![mark\_eu](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ebca7d/32.png) [@mark\_eu](https://discuss.pytorch.org/u/mark_eu)\
**Post date:** [March 24, 2022, 10:58pm UTC](https://discuss.pytorch.org/t/help-needed-cuda-out-of-memory/146915/6 "2022-03-24T22:58:07Z")

</div>

Thank you, @ptrblck, for your advice.  
I changed the model into:

```auto
class CNN(nn.Module):
    def __init__ (self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim,
                 dropout, pad_idx):

        super(). __init__ ()

        self.embedding = nn.Embedding(vocab_size, embedding_dim, pad_idx)
        '''
        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1,
                                              out_channels = n_filters,
                                              kernel_size = (fs, embedding_dim))
                                    for fs in filter_sizes
                                    ])
        '''
        self.conv_0 = nn.Conv2d(in_channels = 1,
                                out_channels = n_filters,
                                kernel_size = (filter_sizes[0], embedding_dim))

        self.conv_1 = nn.Conv2d(in_channels = 1,
                                out_channels = n_filters,
                                kernel_size = (filter_sizes[1], embedding_dim))

        self.conv_2 = nn.Conv2d(in_channels = 1,
                                out_channels = n_filters,
                                kernel_size = (filter_sizes[2], embedding_dim))
        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)

        self.dropout = nn.Dropout(dropout)

    def forward(self, text):
        text = text.permute(1, 0) 
        embedded = self.embedding(text)
        embedded = embedded.unsqueeze(1)

        conved_0 = F.relu(self.conv_0(embedded).squeeze(3))
        conved_1 = F.relu(self.conv_1(embedded).squeeze(3))
        conved_2 = F.relu(self.conv_2(embedded).squeeze(3))
        '''
        conved_n = [batch size, n_filters, sent len - filter_sizes[n] + 1]
        '''
        pooled_0 = F.max_pool1d(conved_0, conved_0.shape[2]).squeeze(2)
        pooled_1 = F.max_pool1d(conved_1, conved_1.shape[2]).squeeze(2)
        pooled_2 = F.max_pool1d(conved_2, conved_2.shape[2]).squeeze(2)
        '''
        pooled_n = [batch size, n_filters]
       '''
        cat = self.dropout(torch.cat((pooled_0, pooled_1, pooled_2), dim = 1))

        return self.fc(cat)

```

However, I still get CUDA out of memory error. Do you have any other suggestions?  
Thank you

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 24, 2022, 11:21pm UTC](https://discuss.pytorch.org/t/help-needed-cuda-out-of-memory/146915/7 "2022-03-24T23:21:53Z")

</div>

There wouldn’t be any difference in your code as you are still storing all intermediate activations.  
Try this:

```python
pooled_0 = F.max_pool1d(F.relu(self.conv_0(embedded).squeeze(3)))

```

---

<div class="post-metadata">

**Author:** ![mark\_eu](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ebca7d/32.png) [@mark\_eu](https://discuss.pytorch.org/u/mark_eu)\
**Post date:** [March 25, 2022, 9:49am UTC](https://discuss.pytorch.org/t/help-needed-cuda-out-of-memory/146915/8 "2022-03-25T09:49:54Z")

</div>

Thank you, @ptrblck, for additional help.  
My model now looks like this:

```auto
import torch.nn as nn
import torch.nn.functional as F

class CNN(nn.Module):
    def __init__ (self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim, 
                 dropout, pad_idx):
        
        super(). __init__ ()
        
        self.embedding = nn.Embedding(vocab_size, embedding_dim, pad_idx)
        '''
        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1, 
                                              out_channels = n_filters, 
                                              kernel_size = (fs, embedding_dim)) 
                                    for fs in filter_sizes
                                    ])
        '''
        self.conv_0 = nn.Conv2d(in_channels = 1, 
                                out_channels = n_filters, 
                                kernel_size = (filter_sizes[0], embedding_dim))
        
        self.conv_1 = nn.Conv2d(in_channels = 1, 
                                out_channels = n_filters, 
                                kernel_size = (filter_sizes[1], embedding_dim))
        
        self.conv_2 = nn.Conv2d(in_channels = 1, 
                                out_channels = n_filters, 
                                kernel_size = (filter_sizes[2], embedding_dim))
        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)
        
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, text):

        text = text.permute(1, 0)
        embedded = self.embedding(text)
                
        
        embedded = embedded.unsqueeze(1)
        
        
        conved_0 = F.relu(self.conv_0(embedded).squeeze(3))
        conved_1 = F.relu(self.conv_1(embedded).squeeze(3))
        conved_2 = F.relu(self.conv_2(embedded).squeeze(3))
            
        '''
        pooled_0 = F.max_pool1d(conved_0, conved_0.shape[2]).squeeze(2)
        pooled_1 = F.max_pool1d(conved_1, conved_1.shape[2]).squeeze(2)
        pooled_2 = F.max_pool1d(conved_2, conved_2.shape[2]).squeeze(2)
        '''
        pooled_0 = F.max_pool1d(F.relu(self.conv_0(embedded).squeeze(3)))
        pooled_1 = F.max_pool1d(F.relu(self.conv_1(embedded).squeeze(3)))
        pooled_2 = F.max_pool1d(F.relu(self.conv_2(embedded).squeeze(3)))
      
        
        cat = self.dropout(torch.cat((pooled_0, pooled_1, pooled_2), dim = 1))

            
        return self.fc(cat)

```

Now, I don’t know if I get Cuda out of memory since I get the error:

```auto
TypeError: _max_pool1d() missing 1 required positional argument: 'kernel_size'

```

Do you have a suggestion on how to solve it?

Thank you

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 25, 2022, 6:17pm UTC](https://discuss.pytorch.org/t/help-needed-cuda-out-of-memory/146915/9 "2022-03-25T18:17:39Z")

</div>

The `F.max_pool1d` operations are missing the kernel size and you are also unnecessarily computing and storing `conved_X`.

---

<div class="post-metadata">

**Author:** ![mark\_eu](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ebca7d/32.png) [@mark\_eu](https://discuss.pytorch.org/u/mark_eu)\
**Post date:** [March 25, 2022, 6:31pm UTC](https://discuss.pytorch.org/t/help-needed-cuda-out-of-memory/146915/10 "2022-03-25T18:31:48Z")

</div>

Thank you, @ptrblck,  
I hope this could be that improvement:

```auto
class CNN(nn.Module):
    def __init__ (self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim, 
                 dropout, pad_idx):
        
        super(). __init__ ()
        
        self.embedding = nn.Embedding(vocab_size, embedding_dim, pad_idx)
        '''
        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1, 
                                              out_channels = n_filters, 
                                              kernel_size = (fs, embedding_dim)) 
                                    for fs in filter_sizes
                                    ])
        '''
        self.conv_0 = nn.Conv2d(in_channels = 1, 
                                out_channels = n_filters, 
                                kernel_size = (filter_sizes[0], embedding_dim))
        
        self.conv_1 = nn.Conv2d(in_channels = 1, 
                                out_channels = n_filters, 
                                kernel_size = (filter_sizes[1], embedding_dim))
        
        self.conv_2 = nn.Conv2d(in_channels = 1, 
                                out_channels = n_filters, 
                                kernel_size = (filter_sizes[2], embedding_dim))
        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)
        
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, text):

        text = text.permute(1, 0) 
        embedded = self.embedding(text)
               
        embedded = embedded.unsqueeze(1)

        conved_0 = F.relu(self.conv_0(embedded).squeeze(3))
        conved_1 = F.relu(self.conv_1(embedded).squeeze(3))
        conved_2 = F.relu(self.conv_2(embedded).squeeze(3))
            
        
        pooled_0 = F.max_pool1d(conved_0, conved_0.shape[2]).squeeze(2)
        pooled_1 = F.max_pool1d(conved_1, conved_1.shape[2]).squeeze(2)
        pooled_2 = F.max_pool1d(conved_2, conved_2.shape[2]).squeeze(2)

        
        cat = self.dropout(torch.cat((pooled_0, pooled_1, pooled_2), dim = 1))

            
        return self.fc(cat)

```

I again get the error:

 ![convolutions - error cuda](https://discuss.pytorch.org/uploads/default/original/3X/b/e/be759fa4347382271e7fc75e405ff5acc05a9587.jpeg)

I lowered batches to 1, but that didn’t help. @ptrblck, do you have any idea what else I could do to solve this problem?

Thank you

---

<div class="post-metadata">

**Author:** ![mark\_eu](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ebca7d/32.png) [@mark\_eu](https://discuss.pytorch.org/u/mark_eu)\
**Post date:** [March 28, 2022, 12:51pm UTC](https://discuss.pytorch.org/t/help-needed-cuda-out-of-memory/146915/11 "2022-03-28T12:51:24Z")

</div>

Does anyone have any additional idea on how to solve this issue? @ptrblck, do you have any further suggestions?

Thank you