Hi,
I’m working on customizing the 70B llama 2 model for my specific needs. Can you provide information on the required GPU VRAM if I were to run it with a batch size of 128? I assumed 64 GB would be enough, but got confused after reading this post. How to deal with excessive memory usages of PyTorch? - vision - PyTorch Forums
Thanks