Several models loaded but just one at a time in GPU

deployment
3

I have find a better approach @bonzogondo :

models = [...] # define a list of all models on the CPU

input = ... # get your input
for model in models:
    model.to('cuda')
    pred = model(input)
    #make something with pred
    del pred
    model.to('cpu')

They key fact is that you need to delete predictions variable if you don’t wank cuda memory issues

1 Like