Multi GPU Training

1

Hi, I have a model that requires 14 GB to train. However, I have access to 12 GB GPU nodes. If I use two nodes (that is 24 GB) still I am getting not enough memory CUDA error. Can you help me overcome this error?

2

If your model cannot fit into single GPU you would need to do model parallelism