A small tensor occupy 600MiB on GPU?

I just put a small 3*4 tensor to GPU0, why the memory on GPU0 increases
from 0 to 600 so rapidly? It’s too high.How to compute it?(just forget the GPU1 , I am using it.) Is there anything wrong with my server?

That’s probably the initialization of CUDA and CuDNN.

what should I do? modify my code or do something on my server?

No, it is always going to consume that bare minimum. Atleast that’s what I have noticed.