The memory footprint grow irregularly with the increased of vision prompts

I hope this message finds you well. I am currently exploring the shallow VPT method as proposed in this GitHub link and would like to experiment with different numbers of inserted prompts. Interestingly, I have observed that the peak memory footprint grows irregularly with the increase of visual prompts, with no significant increase from 2 to 3. I have shared the results below. I’m curious to understand which features in PyTorch contribute to this outcome. Your insights would be greatly appreciated.

tokens number 0 1 2 3 4 5 6 7 8 9 10
Memory (MB) 818.9 824.1 827.1 827.1 828.6 831.1 831.9 836.1 836.9 838.6 842.4
Extra (MB) 0 5.2 8.2 8.2 9.7 12.2 13 17.2 18 19.7 23.5

Not an expert in this specifically but typically the CUDA caching allocator will sometimes request more memory than it needs and manage that memory A guide to PyTorch’s CUDA Caching Allocator | Zach’s Blog