I’m doing mixed-precision training (from the native amp in pytorch 1.6) on feedforward neural networks. Both the training time and memory consumed have increased as a result.
The GPU is RTX 2080Ti. I tried to have all of the dimensions in multiples of 8 as well.
The training time is less important to me, I mainly want to decrease the memory footprint as much as possible since I’m using large feedforward neural networks only.