NoiseTunnel: CUDA out of memory when trying out

nareshr8 · March 27, 2020, 4:49am

I was trying out captum. I am getting CUDA out of memory when I try NoiseTunnel option in captum.

Can someone help in solving where I am going wrong…

ptrblck · March 27, 2020, 4:55am

Your device might not have enough memory and thus you are seeing the OOM issue.
Could you explain, what noise_tunnel does and could you try to lower the batch size or number of samples you are passing to this method?

nareshr8 · March 27, 2020, 5:01am

I am using GeForce GTX 1070 Ti, which has 8 GB of memory. I was just trying out the example here. I am using just one image.

ptrblck · March 27, 2020, 5:07am

Could you check the memory usage via nvidia-smi and make sure the device is empty and stop other processes, if necessary?
I haven’t tried the code, so I’m not sure how large the expected memory footprint is.

nareshr8 · March 27, 2020, 5:12am

I saw that it as using just around 300-400 MB before running this line. And on execution, it went out of memory.

Narine · April 4, 2020, 1:02am

@nareshr8, you can reduce n_samples to limit the number of perturbed examples and n_steps to reduce the number of integral approximation steps.
With our recent improvements (the PR got merged a couple of days ago) you should be able to adjust internal_batch_size and run IG for very large number of steps.

nareshr8 · April 4, 2020, 1:19pm

Thanks a lot @Narine it worked for me. Thanks.