I’m using a bash script to do grid-search for my pytorch code.
Something like as follows:
for lr in 0.1 0.01 0.001 do for wd in 0.1 0.01 0.001 do # call pytorch script for the current setting of hyperparams python my_pytorch_script.py --lr=$lr --wd=$wd done done
I noticed that, sometimes, a process finishes without freeing the GPU memory. It eventually causes the GPU to go OOM even though I execute each program sequentially.
How can I fix this?