GPU memory issue on UNET

Not necessarily. Depends on your train_loader. I’d recommend you, to load them to CPU-RAM inside your loader and just push them to GPU inside the train loop.