How can I switch between two batch sizes for train_loader during training?

vision
1

I hope to switch between two batch sizes for training data during data, for different epochs.
For example,

if epoch%2==0:    train with batch_size 4 (i.e. this is a for loop to grab each batch of size 4)
else:  train with batch_size 8 (i.e. this is a for loop to grab each batch of size 8)

I tried to define two train_loaders at the very beginning like this:

train_small = torch.util.data.Dataloader(.., batch_size=4 ,,,)
train_large = torch.util.data.Dataloader(.., batch_size=8 ,,,)
if epoch%2==0:
    train_loader =  train_small
else 
    train_loader = train_large
for input, target in tqdm(train_loader ...):
    do some training.

But got the “out of cuda memory” error. However, if I just use “train_large” for all epochs, it is ok.
Is this the correct approach to switch between 2 batch sizes during training?

The reason I am doing this is, different training epoch consumes different size of memory, so I want to use smaller batch for even epochs but a larger batch size for odd epochs. Could someone provide some guidance on how to achieve that? Thanks.

2

Hi

You tell very little about your model architecture and data structure. However, in even epoch cycles, you could copy the part of the batch you want and delete from memory the other parts before you start any computation. This way- when you start computing for the batch, memory is of the size you want. It will not cover the whole dataset but will indicate if a smaller batch helps even epochs

3

Thanks for the response. Can you provide some exemplary code to achieve “in even epoch cycles, you could copy the part of the batch you want and delete from memory the other parts before you start any computation”? I never manually handled memory before.

4

so what I think you should do:
if epoch%2 == 0:
data_small = data[0:new_size]
del data

give it a try and let me know what happens

I think that in your previous attempt you created a confusion between the data loaders because they are data loaders from the same dataset. in the method suggested here - you use only the large data loader, but use only a portion of the batch each even epoch. So if the suggestion above works, it will skip part of the data. This can be fixed later - maybe use only the small batch data loader taking 4 batches every ‘odd’ epoch. We can see after you try the above