SVD grad values are 'NaN'

1

Hi all,
when dealing with matrices with number of rows much greater than number of columns (X10)
I’m receiving NaN grads for SVD

when I use matrix size of 256X15 everything is fine
when I use 4096X100 I get NaN (this is the size I need)
any idea why?

2

I have a similar problem. I also get NaN in the gradient. My matrix has the size of [2028, 64]. Please post the solutions if you fixed it.

3

There are a few solutions

1- You can add small amount of noise to your matrix

2- Since nan are for numbers that are very large or small you can use torch.nan_to_num

3- you can use linalg.svd(X.cpu().detach().numpy(), full_matrices=False, lapack_driver=“gesvd”), which I believe still have problem but better than torch.svd

4

The gradient of an SVD decomposition is only defined if the singular values aren’t arbitrarily close together. There’s more detail about this shown in the documentation here.