Multi threading memory leak in inference

I am using Flask with Pytorch. Every new thread take 20 30mb memory and it is not released after thread is completed . If I do 100 threads in flask then it will take 20mb on 1st 100 calls. How to release that memory ? tried empty_cache, torch.no_grad() but got no luck.