Matrix ill-conditioning

I have derived a gradient for backward propagation which requires matrix inversion. I am using moore-penrose inverse in torch for matrix inversion. The gradient decreases the loss to some value after that it gives an error “matrix is ill-conditioned or has too many repeated singular values”. I changed the matrix precision from float 32 to float 64 which improved the results but still after decreasing the loss to some extent it gives the matrix ill-conditioning error.

Can anyone help?
Thanks