CUDNN_STATUS_NOT_SUPPORTED error occurs when apply autograd.grad to compute high-order differentiation

Hi, I’m working on my project involving the great high-order differentiation operator. It works fine under CPU stride, however, when I try to work on GPU strides, I encountered the following error

RuntimeError: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

I’m working with python3.6.1, torch 0.2.0_4, CUDA Version 8.0.44 and cudnn 6021, which I think are all updated. I track back the error, and it shows the error shows up whenever I use conv2d layer.

I tried torch.backends.cudnn.enabled=False to get rid of the error but then no acceleration happens seems.

Some other solution like torch.backends.cudnn.benchmark=True is not working in my case.

As for now, I’m not able to find a way to resolve this, would anyone help me to figure it out? thank you in advance,

I created the following demo code to show my issue, please let me know if you need anything more from me :blush:

import torch
import torch.nn as nn
from torch.autograd import Variable, grad
import as Data
import torchvision

train_data = torchvision.datasets.MNIST(

train_loader = Data.DataLoader(
    dataset=train_data, batch_size=50, shuffle=True, num_workers=2)

class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Sequential(nn.Conv2d(1, 16, 5, 1, 2))
        self.out = nn.Linear(16 * 28 * 28, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = x.view(x.size(0), -1)
        output = self.out(x)
        return output, x

cnn = CNN()

loss_func = nn.CrossEntropyLoss()

for step, (data, label) in enumerate(train_loader):
    input = Variable(data).cuda()
    target = Variable(label).cuda()

    output = cnn(input)[0]
    loss = loss_func(output, target)

    params = cnn.parameters()
    g = grad(loss, params, create_graph=True)

    g_sum = 0
    for g_para in g:
        g_sum += g_para.sum()

    params = cnn.parameters()
    hv = grad(g_sum, params, create_graph=True)


The error message is as following in my setting:

Traceback (most recent call last):
  File "", line 67, in <module>
    hv = grad(g_sum, params, create_graph=True)
  File "/root/miniconda3/lib/python3.6/site-packages/torch/autograd/", line 153, in grad
    inputs, only_inputs)
RuntimeError: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

Looks like a bug, we’ll get this fixed. Opened an issue here to track this:

Hi, this issue also happens for Conv1d, is that also fixed?

Could you post the nn.Conv1d setup, the input shape, as well as the CUDA and cudnn version so that we could reproduce it, please?