Backward CNN with multiple output

autograd
1

I’m now reimplementing the “Holistically-nested Edge Detection”(arXiv) with pytorch, which has multiple “side-outputs”.

So how can I backward CNN with multiple outputs ? This thread suggested to use torch.autograd.backward([x1, x2], [grad_x1, grad_x2]), but I run into error with this solution, following is my error message:

Traceback (most recent call last):
  File "main.py", line 68, in <module>
    train()
  File "main.py", line 63, in train
    torch.autograd.backward(outputs, grads)
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/__init__.py", line 98, in backward
    variables, grad_variables, retain_graph)
RuntimeError: CUDNN_STATUS_MAPPING_ERROR

My code can be summarized as follow:

outputs = model(data)
loss, grads = some_numpy_tricks(outputs)
torch.autograd.backward(outputs, grads)
print "now loss=%f"%loss

In which outputs is a list containing multiple outputs of the model, and grads is the list containing gradients of loss w.r.t each model output.

So why did error occur and what’s the correct solution for my case?

2

this seems like a cudnn bug.
Can you give me a way to reproduce this on my side?

You can also try setting this flag (at the beginning of your script) and see if it goes away: torch.backends.cudnn.benchmark=True

1 Like
3

Hi Smth, sorry to bothering you again.

I met the new error:

RuntimeError: dependency not found for N5torch8autograd12ConvBackwardE

I have been working on this for long but cannot solve the problem.

I have committed the code to Gitlab, could you please help me to solve this problem?

4

which version of pytorch are you on? print(torch.__version__), how did you install pytorch (conda or pip)? and what OS are you on?

5

Thanks for your reply, my environments are as follow:
PyTorch:

In [2]: print torch.__version__
0.1.12_2

Installed via pip following the official instructions.
OS:

uname -a
Linux server01 4.4.0-72-generic #93~14.04.1-Ubuntu SMP Fri Mar 31 15:05:15 UTC 2017 x86_64 x86_64 x86_64 
GNU/Linux

CUDA:

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Sun_Sep__4_22:14:01_CDT_2016
Cuda compilation tools, release 8.0, V8.0.44

The error mentioned before occurs both on CPU and GPU mode, so it seems not to be a CUDA problem.

6

I’m really hoping that after you upgraded to v0.2.0, this issue has gone.

7

OK, I will update to v0.2 and retry.

Thank you.

8

@zeakey Hi, have you finished the reimplementing? Will you public your code?

9

Though don’t know why, it did work. Thanks so much.