Create/edit PyTorch tensor using OpenGL?


(Rodrigo Vargas) #1

In order to create polygonal masks I’m currently using Pillow’s ImageDraw to draw them. Then, I can get the corresponding numpy arrays and upload to GPU. But I’m thinking about creating them directly on the GPU using OpenGL, via, say, pyglet or glumpy. I found somewhere else how to pass PyTorch tensors to CuPy using data_ptr() and the current CUDA stream, and I wonder whether something along those lines can be used to “draw” to a PyTorch tensor using OpenGL. Does anyone know how to do that?


(colesbury) #2

You can, but it’s going to require a bunch of work including C/C++ code.

  1. You need render to a texture using pyglet or glumpy. Get the texture ID (an int)

  2. Use the CUDA OpenGL API to register the texture: https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__OPENGL.html

  3. Use the CUDA Graphics API to bind the resource to a pointer. https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__INTEROP.html#group__CUDART__INTEROP

  4. Either cudaMemcpy that pointer into a Tensor’s data_ptr() or use torch::from_blob to wrap it as a Tensor.

You’ll also need to call the appropriate unregister functions in destruction and make sure that the texture is in a matching format as the Tensor (i.e. float32 vs. int8)


(Rodrigo Vargas) #3

Could one avoid C/C++ coding for this with PyCUDA?


(colesbury) #4

I don’t think PyCUDA exposes the necessary functions. You might be able to use the Python ctypes library to call the functions, but it’s probably no easier than writing the actual C/C++ code.


(Andrew Pouliot) #5

You might be interested in this gist, which uses a CUDA memcpy via PyCUDA to go the other direction. Presumably you could use the same technique to go from a texture to PyTorch.