I noticed that training a ResNet-50 with AMP on my new laptop with RTX3070 takes much more GPU memory than without AMP. It is not a code issue because I am able to run the same code on a workstation with an Nvidia Tesla p100 with the opposite result.
Ubuntu 20.04 with Kernel 5.8 (and 5.11)
Latest Nvidia drivers installed with Ubuntu GUI software and updates (CUDA 11.2)
command numba -s returns no error regarding CUDA install. Same for nvidia-smi.
Training without AMP: 3.9 GB VRAM
Training with AMP: 7.4 GB VRAM
GPU memory consumption is stable during training
I installed Pytroch with:
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia
(Also tried Pytorch preview 1.9)
GPU: Nvidia tesla p100
Training without AMP: 3.5 GB VRAM
Training with AMP: 2.5 GB VRAM
Do you have any idea why I have this strange behavior on my laptop?
Thanks in advance