I am doing k-fold cross-validation with a simple model. After a loop finishes 10 iterations (for 10-folds validation), and the program is supposed to terminate, I get the following error:
AttributeError: ‘NoneType’ object has no attribute ‘cudnnDestroyDropoutDescriptor’
and
AttributeError: ‘NoneType’ object has no attribute ‘cudnnDestroy’
Any tip, why I am getting the error?
Source code:
for train_idx, dev_idx in skf.split(corpus.data, corpus.labels):
train_data, dev_data = numpy.array(corpus.data)[train_idx], numpy.array(corpus.data)[dev_idx]
train_corpus, dev_corpus = data.Corpus(args.tokenize), data.Corpus(args.tokenize)
train_corpus.data = train_data
dev_corpus.data = dev_data
model = BCN(dictionary, embeddings_index, args)
optim_fn, optim_params = helper.get_optimizer(args.optimizer)
optimizer = optim_fn(filter(lambda p: p.requires_grad, model.parameters()), **optim_params)
best_acc = 0
if args.cuda:
model = model.cuda()
obj = train.Train(model, optimizer, dictionary, args, best_acc)
obj.train_epochs(train_corpus, dev_corpus, args.start_epoch, args.epochs)
print('\nTESTING : K = ' + str((k + 1)))
test_batches = helper.batchify(test_corpus.data, args.batch_size)
accuracy, f1 = evaluate(model, test_batches, dictionary)
print('accuracy: %.2f%%' % accuracy)
print('f1: %.2f%%' % f1)
test_acc.append(accuracy)
test_f1.append(f1)
k += 1
I get the error when for loop finishes its’ execution.
Edit:
I found a solution (as soumith mentioned here: Error following tutorial (PyTorch with CUDA 7.5)). Before running my program, if I run the following two commands:
unset LD_LIBRARY_PATH
unset LD_PRELOAD
Then it doesn’t throw the error. But each time, when I log in to the server, I need to run the above two commands. Does anybody have any permanent remedy to this problem?
PROBLEM SOLVED.