Sudden GPU memory increase before training

I am trying to train a UNet, right before training, the process stuck and sudden GPU memory increase is observered. Then, when the training starts, the GPU memory drops and keeps stable. How can I resolve the problem? thanks!