Use torch.svd with complex matrix on GPU

#1

Hi all, I am trying to calculate the SVD of some complex matrices. It worked fine on the CPU, but when it comes to the GPU, some errors happened, here is the snippet of my code and its corresponding output:

print("M_final is: ", M_final)
print(M_final.dtype)
print(M_final.device)
SVD_result = torch.svd(M_final)
print(SVD_result)

......Output:......

M_final is:  tensor([[ 0.2688+1.2463e-05j, -0.2223+1.0665e-01j],
        [-0.1943-1.3049e-01j,  0.0545-3.9980e-01j],
        [-0.5253-5.2017e-01j,  0.1036+3.6822e-01j],
        [-0.2718-4.8315e-01j, -0.2331-5.5959e-01j],
        [ 0.0798+1.1285e-01j, -0.2115-4.6674e-01j]], device='cuda:0')
torch.complex64
cuda:0
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-129-4a7caa855681> in <module>
     30 print(M_final.dtype)
     31 print(M_final.device)
---> 32 SVD_result = torch.svd(M_final)
     33 print(SVD_result)

RuntimeError: "svd_cuda" not implemented for 'ComplexFloat'

Does this mean that the torch.svd does not support the complex tensor on the GPU? I checked some posts and issues, it seemed that this functionality is actually supported on GPU: https://github.com/pytorch/pytorch/pull/42738

Can anyone give some suggstions on this? Thanks in advance!

#2

Hello Yuchen!

If you can tolerate the risks of living on the “bleeding edge,” it appears
that the nightly build (version 1.8.0) now has complex svd on the gpu:

>>> import torch
>>> torch.__version__
'1.8.0.dev20201203'
>>> torch.svd (torch.randn ([2, 5], dtype = torch.cfloat, device = 'cuda:0'))
torch.return_types.svd(
U=tensor([[-0.9239+0.0000e+00j, -0.3826+1.1088e-08j],
        [ 0.0543+3.7870e-01j, -0.1311-9.1457e-01j]], device='cuda:0'),
S=tensor([2.3032, 1.6723], device='cuda:0'),
V=tensor([[ 0.3954-0.1435j, -0.2532+0.1638j],
        [ 0.0011+0.3421j,  0.1400-0.3463j],
        [ 0.0546-0.4828j, -0.3884-0.4571j],
        [ 0.1831+0.3384j, -0.4973-0.1605j],
        [-0.2001-0.5309j,  0.0373-0.3678j]], device='cuda:0'))

Best.

K. Frank

#3

Hi Frank!

Nice to see you here again :slight_smile: and thank you very much! Let me try it and see what happens.

Regards,
Yuchen

#4

Hi Frank,

I tried the nightly version and yes, this version supported the (forward) computation of SVD for comlex matrix, but when I implemented a customized layer which included this complex matrix SVD cmputation, and put this layer inside my model, during the training time the following error occurred:

RuntimeError: svd does not support automatic differentiation for outputs with complex dtype.

It seems like 1.8 verson still does not support the aotugrad for SVD when complex matrix is used, any suggestions or should I wait Pytorch develop team to improve this functionality?

Regards,
Yuchen

#5

Hello Yuchen!

I don’t know of any quick fix for this.

Yes, I think waiting is probably the best option. Complex tensors are
currently a work in progress in pytorch.

As a further note, I’ve been thinking about this some, and I don’t fully
understand how autograd ought to work with complex tensor functions.
If I can get my thoughts sorted out on this I will likely post something,
but right now I am very confused by the whole thing.

Best.

K. Frank

#6

Ok I see it, thank you Frank!