Libtorch C++ convert a Tensor to cv:Mat (single channel

C++
1

Hi, big masters and huge Gods,

I wanna ask a question hope anyone open this link can give me a help which is convert a Tensor to cv::Mat.

I have a Tensor outputs from my model and it’s [1, 19, 32, 43] shape, which is, batch_size, 19 instances, and 32x43 is the featuremap size.

I have it’s type: CPUFloatType. And I want converts it into cv::Mat, here is what I do:

 cv::Mat one_mat(heatMapsTensor.size(2), heatMapsTensor.size(3), CV_32FC1);
std::memcpy((void *) one_mat.data, one_heat_map.data_ptr(), sizeof(torch::kFloat32) * one_heat_map.numel());

heatMapsTensor is the tensor described above. one_mat is what I need to got.

But I got the mat incorrect. what did I wrong?

I suspect 2 maybe mistake:

  • the type maybe not float32, but I do not know which it is;
  • the mat shape is not wrong

Any body could help me out?

2

Since you are using the tensor.data_ptr() you would not use ...sizeof(torch::kFloat32) * numel... but rather ...sizeof(float) * numel...

Try something like:

std::memcpy(cv_pointer, tensor.data_ptr(), tensor.numel() * sizeof(float));

Assuming you are using floats.
You may also want to have a look (Convert torch::tensor to cv::Mat . from @mhubii )

1 Like
3

thanks for you reply!!

I changed into this:

	  cv::Mat one_mat(heatMapsTensor.size(1), heatMapsTensor.size(2), CV_32FC1);
	  std::memcpy(one_mat.data, one_heat_map.data_ptr(), sizeof(float) * one_heat_map.numel());

This time, the cv::Mat values seems normal, but result still not right.

what I want , is convert a tensor in multi dimension: [19, 32, 46] into a vector, length is 19, and every member in vetor is cv::Mat with size of 32,46.

Now, dims is right, the issue is the value converted into cv::Mat is always not right, it is not the same value in tensor.

1 Like
4

Ahh,
then those are two kinds of problems.
What you can do is loop over the first dimension and slice the tensor and then do std::mmcpy

So something like this

std::vector<cv::Mat> list_of_mat;
for (int i=0; i < tensor.size(0); i++) {
// tensor slicing
// std::memcpy to dummy
list_of_mat.pushback(dummy)
}
1 Like
5

This is actually what I am doing for now:

std::vector<cv::Mat> heatMaps(heatMapsTensor.size(0));

	for (size_t i = 0; i < heatMaps.size(); i++) {
	  torch::Tensor one_heat_map = heatMapsTensor[i];
	  cv::Mat one_mat(heatMapsTensor.size(1), heatMapsTensor.size(2), CV_32FC1);
	  std::memcpy(one_mat.data, one_heat_map.data<float>(), sizeof(float) * one_heat_map.numel());
	  heatMaps[i] = one_mat;
	}

I just saw this usage tensor_a.data<float>() which seems a clue so I changed to this but still not right.

I am trying another thread:

get the tensor pointer say float* p, then copy it’s data to cv::Mat. However, how should I get the point of a tensor ? using tensor_a.data_ptr()? But that is (void*) not what I need float*

1 Like
6

From cv2 to TORCH:

 cv::cvtColor(frame, frame, CV_BGR2RGB);
 frame.convertTo(frame, CV_32FC3, 1.0f / 255.0f);
 auto input_tensor = torch::from_blob(frame.data, {1, frame_h, frame_w, kCHANNELS});
 input_tensor = input_tensor.permute({0, 3, 1, 2});

From TORCh to cv2:

input_tensor = input_tensor.to(at::kCUDA);
                torch::Tensor out_tensor = module->forward({input_tensor}).toTensor();

                out_tensor = out_tensor.squeeze().detach().permute({1, 2, 0});
                out_tensor = out_tensor.mul(255).clamp(0, 255).to(torch::kU8);
                out_tensor = out_tensor.to(torch::kCPU);
                cv::Mat resultImg(frame_h, frame_w, CV_8UC3);
                std::memcpy((void *) resultImg.data, out_tensor.data_ptr(), sizeof(torch::kU8) * out_tensor.numel());
8 Likes
7

this really works!!!

1 Like
8

A million thanks to you!
Would it be possible to also post how to get CV_32C3 format instead of CV_8UC3?
I cant get it working, im getting artifacts

1 Like
9

It works! I think the process" out_tensor = out_tensor.to(torch::kCPU) " is crucial , I am right??
only we must put the output tensor from GPU to CPU, then we can convert the data to Mat?!

10

Yes, indeed. Since the OpenCV MAT does not reside on the GPU.

11

Would it be possible to load the image directly in RGB? It slows down a bit my inferences… :slight_smile:

13

Try other libraries as Pangolin

14

Hi Shlomo,may I ask a question, thank you in advance.
What if my out_tensor is two segmentation maps（float） and I want to convert it to CV_8UC1 mat?

    torch::Tensor output = model.forward({ tensor_image }).toTensor();
    output.print();  //  ==>1x2x256x256
    //apply softmax layer
    torch::Tensor probs = torch::softmax(output, 1)[0];
    probs.print(); //  ==> 2x256x256

Here,I got two probabilities map for a semantic segmentation task.One is the background class and the other is the object class.Now I want to convert from torch to cv2,simply I just use this one line code,

Mat seg_map(256, 256, CV_32FC1, probs[1].data_ptr());

However, I found there are some difference from model inference in Python.Is there a better way to copy tensor data to mat?
Thank you for your attention.

15

I would offer a couple of notes here in order for people to perhaps get a bigger picture. This code is great and simply works (very difficult to find working code for libtorch in general PERIOD).

A usual workflow would include PyTorch (where most people would develop the model and train it). After the model has been trained, you need to dump it to the CPU (that is, the RAM, not VRAM) and export it using tracing, which is part of TorchScript. This process is basically serializing your model so that it can be loaded both in Python as well as in C++.

A very important aspect is how you train your model. I was just playing around with SRCNN (among the oldest super resolution models out there) and it dawned on me that I have trained it with single channel images and not 3. So accordingly when you are loading an image with OpenCV you will have to consider that too.