CUDA out of memory error when calling the train script in a loop from another python script

Hi all,

I’m trying to run the same pytorch training script with different arguments (argparse) from another python script. I’m using os.system() for the same.

Here’s what I’m trying to do - = > the script which contains the train-loop. => the file which runs the train script in a loop.

for hp in hyperparams:
    os.system(f"CUDA_VISIBLE_DEVICES=1 python --arg1 hp")

A few models get trained but I eventually end up getting a CUDA out of memory error. My guess is that the GPU memory is not being cleared after every loop. What can I do to mitigate this?

Does it train completely for the first set of hyperparams in the for loop?

Yep. For instance, if there were 10 models, it will successfully train 8 and then give a CUDA error for 9 and 10.