Perhaps someone can help me with this. I’ve written a small network and I’d like to pass the output of this tensor to OpenCV to do some further image processing and pass the output of that back to a tensor which then is used in calculating the loss. Converting the tensors to cv::Mats and vice versa is not the problem, however, the weights don’t seem to update in the network and therefore doesn’t train. The gist of it is like so:

Try to create a new copy by detaching the original tensors and then convert them to cv2, do your processing and then add them to original tensors. See if it works.

I’ve attempted your suggestion but it doesn’t solve the issue, although I’m not sure what you meant by “adding the result to the original tensors”.

torch::Tensor prediction = net->forward(img_tensor);
// create a new instance of the tensor by detaching the original tensor
torch::Tensor prediction_cpy = prediction.detach();
cv::Mat prediction_vectors(cv::Size(512, 512), CV_32FC3, prediction_cpy.data_ptr());
// some image processing magic
cv::Mat someOutput = someImageProcessingMagic;
// do you mean add?
prediction.add(torch::from_blob(someOutput.data, { 1, 3, 512, 512 }));
torch::Tensor loss = torch::mse_loss(prediction, target);

I meant inorder to do your opencv operations, you can detach the values from the original graph, do all your stuff with opencv and in the end, you can add those opencv results to your original graph if that is what you want (for example to compute some loss or something else).

I see. Yes, that’s not going to work. The output of the network is a tensor of motion vectors, which then I want to use to warp an image which is then used as a new tensor of a predicted (warped) image and calculate the loss, corresponding to the desired target image. Hope it makes sense. If not, I’m happy to give you a more detailed used case.

So, I had some time to play around with this and the code makes perfect sense. However, during debugging I noticed something strange. The output values from the “grid” tensor don’t match up to the new opencv Mat pointer from the tensor. What makes it even more mysterious is that if the grid tensor is either a 1 or 3 channel tensor then the values match. Here is the code: