I’m looking for a way of reinterpreting a PyTorch tensor as a different dtype of matched size. This kind of operation is provided by NumPy, but as far as I can tell PyTorch only provides casts. (naturally, reinterpreting types is an operation that would only make sense for tensors without gradients).
These kinds of type reinterpretations are an important building block for implementing special functions. It’s quite common to switch from float to integer, do a few manipulations there (e.g. of the mantissa), and then switch back to floats in the end. In the NumPy world, np.view is the operation that enables one to do that. AFAIK it’s not possible with PyTorch. There is no analog of np.view as far as I can see, and directly assigning the dtype yields an error message:
>>> t.dtype = torch.int32
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: attribute 'dtype' of 'torch._C._TensorBase' objects is not writable
If you are writing a C++ extension function, you can use reinterpret_cast<int64_t*>(tensor.data<float>()). But unfortunately I don’t think that this is well supported in Python. Please file a feature request on GitHub. Thanks!
Wonder if feature request has been filed? Another possible usage (for the lack of better option?) - cpu HalfTensor doesn’t support loads of operations, even as basic as index() & index_select(), which are useful for slicing batch from large (regressor) matrix (index_select is also very efficient/fast)
So one workaround is ‘pack’ 2 float16 columns to one float32 (or even 4 float16 to one float64), then select required rows (for next batch), then ‘unpack’ back to float16