Training two different CNN architectures (small vs big) take 20GB of vram while training

[ptrblck]

@DoctorPolygon

but this seems excessive, and also odd, given the model size difference.

You are not accounting the intermediate activations, which might use significantly more memory than the parameters and also fits:

@DoctorPolygon

I also tried batch sizes of ‘1’ and my simple resnet will eat about 2GB of vram, and it does seem to scale linearly from there, ending up at around 18-19GB when back at 8 batches.

This post estimates the memory usage from parameters and forward activations.