CUDA out of memory. Tried to allocate 6.85 GiB (GPU 0; 23.69 GiB total capacity; 9.79 GiB already allocated; 2.73 GiB free; 16.13 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
The message says that the reserved memory is significantly large than the allocated memory, we should handle the fragmentation. What needs to be done to handle the fragmentation?