Is there a way to have pytorch automatically offload portions of a large model to main RAM and load them back onto GPU ram for computations?
You can assign where each part of the model is allocated manually. Moving tons of data would be efficient for training. As far as I know it should be complicated and it’s not designed to be done by default. I don’t know if there is a library to do that but I doubt it.
If you’re trying to offload GPU memory to RAM perhaps you might want to have a look at torch.utils.checkpoint — PyTorch 1.8.0 documentation . Although, it’s not exactly what you’re looking for, this might help reduce the amount of memory you need on the GPU!