Out of memory when I use torch.cuda.empty_cache

This issue sound a bin weird.
Are you able to create a dummy tensor on your GPU before empty_cache raises this OOM error?
How much memory is being used before the call?