I am using a data loader to load batches of data into a DataParallel network. The issue is that my network uses batch normalization and batchnorm errors out when batch size is 1. But with 3 GPUs and a batch size of 15, I have an incomplete batch as the epoch change. This often leads at least one GPU with a batch size of 1. I have tried multiple variations of batch sizes, but the dataset size keeps giving me this problem. I don’t know how to resolve this using the automatic batching in the data loader. Is there an option to pad that last batch with all zeros or something?
you can manually pad the last batch or just drop it using drop_last=True
1 Like