Create/edit PyTorch tensor using OpenGL?

(Rodrigo Vargas) #1

In order to create polygonal masks I’m currently using Pillow’s ImageDraw to draw them. Then, I can get the corresponding numpy arrays and upload to GPU. But I’m thinking about creating them directly on the GPU using OpenGL, via, say, pyglet or glumpy. I found somewhere else how to pass PyTorch tensors to CuPy using data_ptr() and the current CUDA stream, and I wonder whether something along those lines can be used to “draw” to a PyTorch tensor using OpenGL. Does anyone know how to do that?

(colesbury) #2

You can, but it’s going to require a bunch of work including C/C++ code.

  1. You need render to a texture using pyglet or glumpy. Get the texture ID (an int)

  2. Use the CUDA OpenGL API to register the texture:

  3. Use the CUDA Graphics API to bind the resource to a pointer.

  4. Either cudaMemcpy that pointer into a Tensor’s data_ptr() or use torch::from_blob to wrap it as a Tensor.

You’ll also need to call the appropriate unregister functions in destruction and make sure that the texture is in a matching format as the Tensor (i.e. float32 vs. int8)

(Rodrigo Vargas) #3

Could one avoid C/C++ coding for this with PyCUDA?

(colesbury) #4

I don’t think PyCUDA exposes the necessary functions. You might be able to use the Python ctypes library to call the functions, but it’s probably no easier than writing the actual C/C++ code.

(Andrew Pouliot) #5

You might be interested in this gist, which uses a CUDA memcpy via PyCUDA to go the other direction. Presumably you could use the same technique to go from a texture to PyTorch.