How to use Pytorch Tensor object in Opencv without convert to numpy array?

1

I’m trying to develop a text detection application with Pytorch and Opencv in Python.
I can use Pytorch tensor with Opencv like below.

val = y[0,:,:,0].data.cpu().numpy()
cv2.threshold(val , 0.4, 1, 0)

But it takes a lot of time. I need to do this operation by using tensor object. How can I do that?

2

Hi,

Give that you use .cpu(), I guess you have a cuda Tensor?
Unfortunately, I don’t think opencv support gpu? So you will have to move the Tensor back to CPU to use it with opencv.
Note that the conversion to numpy itself is almost free as we share memory with the numpy array.

If you use operations that are available on pytorch, I would advise using pytorch’s gpu version of these ops to keep best performances !

3

I already used .cpu() but I got Expected Ptr<cv::UMat> for argument '%s' error. I know I should GPU version but OpenCV does not support gpu and convert from torch.tensor() numpy array is slowing the process.

So I want to give image to OpenCV as torch.tensor()(CPU version) but I got this error Expected Ptr<cv::UMat> for argument '%s'

4

Hi,

The conversion from the torch Tensor to the numpy array cannot be the slow part as it is doing almost nothing.
It might be the conversion done by openCV from a numpy array to an open cv matrix?

5

I have the same question. In my case, I have a tensor which is the output of my model with requires_grad = True and I want to do some process on it using OpenCV in my custom loss. I’m trying to do it on GPU while keeping the requires_grad = True. Do you have any suggestions? I found kornia and it seems they are adding some OpenCV functions, but not the ones I need.

6

Hi,

If you want to use OpenCV, you won’t be able to use autograd (just like if you want to use numpy).
You will need to write a custom autograd Function to tell the autograd what the backward should be: https://pytorch.org/docs/stable/notes/extending.html

1 Like
7

Thanks. The link was very helpful. I can avoid using opencv at the first step. but there is another problem.
Let me explain my problem a little bit more. The outputs of my network are n (x,y) positions, for a car, and they are in the world coordinate frame. I do some transformation on the output to bring them in the image coordinate frame by using pytorch and kornia and keep the gradient. I wanted to draw some rectangles or ellipses for these points on a mask but for now I just want to consider them as a single point in the mask tensor.
Now I have a target mask (an rgb image with zero and one elements) and want to calculate a metric like IoU between this target mask and my n points which are transformed into image coordinate frame. I wanted to create a mask for these points and easily calculate the IoU with the target mask, but I don’t know how to do it to be able to use backprop and keep gradient. Can I attach the created mask tensor from my points to the graph and do backpropagation?

8

Hi,

Few things that jump to my mind:

  • IoU is not differentiable AFAIK
  • If you create a mask containing binary values, you won’t be able to backprop as gradients only exist for continuous values.
1 Like
9

You are right. I think I need to have a head to output a mask directly. Thanks for the help.