Drawing gpu tensors to screen with pycuda/glumpy - updating depracted code

clbr · September 23, 2021, 5:15pm

Hey there !
I’m currently experimenting around with using cuda and opengl (pycuda/glumpy) to draw tensors to the screen. I’ve found a gist that seems to have implemented exactly what i want to do. The problem is that the project is quite old and it uses a few deprecated apis.

The gist:

gist.github.com

https://gist.github.com/victor-shepardson/5b3d3087dc2b4817b9bffdb8e87a57c4

pytorch-glumpy.py

from contextlib import contextmanager
import numpy as np
import torch
from torch import Tensor, ByteTensor
import torch.nn.functional as F
from torch.autograd import Variable
import pycuda.driver
from pycuda.gl import graphics_map_flags
from glumpy import app, gloo, gl

This file has been truncated. show original

Being new to pytorch and machine learning in general, i was wondering if any of you guy’s have a few hints for me on how to get this working.

The packages i have installed:
python - 3.9.7
pytorch - 1.9.0+cu111
pycuda - 2021.1+cuda114
glumpy - 1.2.0

Since this is only doing inference there are quite a few Variable(state, volatile=True). Variable seems to be deprecated, which is easily solved by doing the processing with torch.no_grad():.

What i’m struggling with is converting the tensor to the proper format of the texture. This is from line 99 onwards. Its mainly line 99 that currently throws RuntimeError: t() expects a tensor with <= 2 dimensions, but self is 3D.

I’d appreciate any help.

Cheers

ptrblck · September 24, 2021, 2:38am

I don’t know what exactly changed in the code execution, but based on the RuntimeError it seems your tensors have now an additional dimension. tensor.t() would transpose a 2D tensor and you should be able to use permute instead on 3D tensors.
However, it might also be a good idea to try to check why the number of dimension changed at all.

clbr · September 24, 2021, 3:43pm

Appreciate the answer. I’m trying to understand how the tensors are being transformed. And how it relates to the outcome of what is being rendered in the end.
Basically we have a texture of the shape [512, 512, 4], meaning [width, height, channels]. I assume the channels are RGBA.
And we want to go from a tensor with the shape [1, 3, 512, 512] and [batch_size, channels, height, width] to the format of the texture so it can be rendered. There is no alpha channel in this yet.

So please correct me if i’m wrong with this. I’d have to perform the following steps:

tensor.squeeze() the tensor to get rid of the batch_size, now we have [3, 512, 512] with [channels, height, width]
tensor.transpose(0, 2) to get [512, 512, 3] [width, height, channels]. My question here: what would’ve been the purpose of calling tensor.t().data on the output again? especially on a 3-dimensional tensor. I couldn’t find anything in the docs or changelogs of pytorch that any of the api’s used before this would’ve transformed the tensor into 2-D.
Add the alpha channel with torch.cat((tensor, tensor[:, :, 0]), 2) with tensor being in the same shape as the output of 2nd step. I still get the error: Tensors must have same number of dimensions: got 2 and 3. The input is obviously 3 dimensional, but tensor[:, :, 0] returns a tensor of the shape [512, 512]. I feel like this is the issue currently. Is there any ‘easy’ way of adding an extra column to all matrices that can represent the alpha channel? I want to go from shape [512, 512, 3] to [512, 512, 4].
Set all values of the alpha channel to 1. tensor[:, :, 3] = 1

Cheers

Edit: Found the answer, can pretty much just do:

    a_tensor = torch.cuda.FloatTensor(h, w, 1)
    tensor = torch.cat((tensor, a_tensor), 2)

Is there a way to do this without having to create a tensor beforehand?

ptrblck · September 24, 2021, 6:31pm

Yes, but I would recommend to specify the dimension via tensor.squeeze(0) to make sure no other dimensions are removed in case they have a size of 1.
I don’t know which PyTorch version was originally used, but I thought t() did not work on >2D data. In any case, use tensor = tensor.permute(1, 2, 0) to create the [H, W, C] tensor and don’t use the deprecated .data attribute.
You are indexing the tensor, which will remove this dimension and you could slice it instead:

tensor = torch.randn(4, 4, 3)
out = tensor[:, :, 0]
print(out.shape)
> torch.Size([4, 4])

out = tensor[:, :, 0:1]
print(out.shape)
> torch.Size([4, 4, 1])

However, I don’t think the alpha channel should just be the red channel from the same image, so you might want to use torch.cat((torch.ones(H, W, 1), tensor), 2) instead.

clbr · September 24, 2021, 8:29pm

Could you explain what you mean with I don't think the alpha channel should just be the red channel from the same image?

Using torch.cat((torch.ones(h, w, 1), tensor), 2) or torch.cat((torch.cuda.FloatTensor(h, w, 1).fill_(1), tensor), 2) to directly initialize the tensor on the gpu. I get something that seems like its completely tinted red.

swapping the tensors passed into torch.cat around like torch.cat((tensor, torch.cuda.FloatTensor(h, w, 1).fill_(1)), 2) gives me the desired output.

I appreciate your help a lot

!

ptrblck · September 24, 2021, 9:42pm

Yes, you are right, the alpha channel would be the last one (RGBA).

In your previous example you are reusing the red channels as the alpha channel, or rather you were concatenating the red channel as the first channel and pushed the original RGB channels back so that you would have this layout [RRGB]=[RGBA]. The alpha channel is not a clone of the red channel, but contains own information.

clbr · September 24, 2021, 10:00pm

Yea that makes sense, thanks!