I’m not sure how to properly ask this question or if I’m missing something. I am training a RNN conditioned with images and text. In my training, I take the generated output from the rnn model, and then generate images from that. I then compute the loss, but I cant run loss.backward() and I dont know how to formulate the code to get the correct loss. Here is what I mean:
criterion = nn.L1Loss() for epoch in range(args.num_epochs): for i, (images, captions, lengths) in enumerate(data_loader): images = Variable(images, volatile=False) captions = Variable(captions,volatile=False) # Forward, Backward and Optimize decoder.zero_grad() encoder.zero_grad() features = resnet152_encoder(images) outputs = lstm_decoder(features, captions, lengths) output_features = outputs_to_image_tensor(outputs) loss = criterion(output_features, images) loss.backward() optimizer.step()
With that code I get error: “RuntimeError: there are no graph nodes that require computing gradients” I think I understand why, it is because in the line " output_features = outputs_to_image_tensor(outputs)" I have done a transformation to the outputs that loses the weights. How can I do back propagation here? Or how do I connect output_features to outputs. What I essentially want it to update outputs weights based off loss of the original images with the outputted images. Does this make sense what Im saying?