I faced a problem when i try to train my dataset on FCOS3D/MMdetection3D.
The error info:
RuntimeError: CUDA out of memory. Tried to allocate 2.83 GiB (GPU 6; 31.75 GiB total capacity; 28.25 GiB already allocated; 2.09 GiB free; 28.60 GiB reserved in total by PyTorch)
I have a question, why the used memory of CUDA:3 is twice as much as others? My guess is that there was a problem during one iteration that caused the memory to not be freed.
Note that not only is the memory usage of device 3 increased, but also device 6 seems to be empty, so you might want to double check, if your code accidentally reuses GPU3 two times and skips GPU6.
RuntimeError: CUDA out of memory. Tried to allocate 2.83 GiB (GPU 6; 31.75 GiB total capacity; 28.25 GiB already allocated; 2.09 GiB free; 28.60 GiB reserved in total by PyTorch)
Some errors happend, so GPU 6 is empty.
