Memory leak during inference

I am trying Vtoonify model but in inference the memory keep increasing after every couple of images. In order to lower the consumption I set a standard size for input image like 320*320 so all tensors are of same size but still it keep consuming the memory. I tried dynamic quantization but results are very bad . Here is my code for quantization:

MODEL_REPO = 'PKUWilliamYang/VToonify'

vtoonify = VToonify(backbone='dualstylegan')

                                    map_location=lambda storage, loc: storage)["g_ema"], strict=False)


quantized_model = torch.quantization.quantize_dynamic(vtoonify, {nn.Linear}, dtype=torch.qint8)

quantized_model.qconfig ='x86')

requires_grad(quantized_model.generator, False)
requires_grad(quantized_model.res, False)
        #"g": g_module.state_dict(),
        #"d": d_module.state_dict(),
        "g_ema": quantized_model.state_dict(),

This should be the result

But after quantization I get this output:

I tried mixed precision and also using with torch.no_grad() but got no luck.

I started with 4GB GPU then moved to 8gb and now 12gb still it manages to eat all of the memory.