Pytorch debugging

#1

I am getting this error while training a Bidirectional attention flow model for question answering.

Starting training …

Starting batch: 0

IndexError Traceback (most recent call last)
in ()
----> 1 train(model, train_dataset)

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
2262 .format(input.size(0), target.size(0)))
2263 if dim == 2:
-> 2264 ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
2265 elif dim == 4:
2266 ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

IndexError: Target 545 is out of bounds.

It seems that there is some problem with the way I set the target values in data preprocessing.
What are the best ways to debug these kinds of errors in pytorch. Are there any tutorials or books I can refer to ?

#2

The first step would be to check the error message, which provides good information for this particular failure: the target contains indices, which are out of bounds in nll_loss.

From a general point I don’t know, if there are any good books to debug Python applications.
If you are using the GPU, rerun the script with CUDA_LAUNCH_BLOCKING=1 python script.py args, as you might often get the wrong lines of code in the stack trace due to the asynchronous CUDA operations.
Alternatively, you could rerun the code on the CPU and should see a better error message.
If you need to debug further, you can use e.g. pdb or gdb for the backend.