DDP mode ,GPU memory always increases during the validation phase, and sometimes it gets stuck

When using DDP mode, at the start of the first epoch during validation, GPU0 suddenly increases its memory usage by 2GB and keeps it that way. GPU1’s utilization sometimes reaches its maximum, causing the validation time to increase. Which part of the code should I pay attention to?yolo