Linear Algebra Gradients

autograd
1

Hi,

I’m looking to back-propagate gradients through a singular value decomposition for regularisation purposes.

I know that I could write my own custom function that operates on a Variable; takes its .data tensor, applies the torch.svd to it, wraps a Variable around its singular values and returns it in the forward pass, and in the backward pass applies the appropriate Jacobian matrix to the incoming gradients.

However, I was wondering whether there was a more elegant (and potentially faster) solution, where I could overwrite the “Type Variable doesn’t implement stateless method svd” Error directly, call Lapack, etc. ?

If someone could guide me through the appropriate steps and source files I need to look at, I’d be very grateful. I suppose these steps would similarly apply to other linear algebra operations which have no associated backward method currently.

Many thanks,
Max

2

You can use scipy/numpy to do write custom forward and backward! It’s quite straightforward.
Check out this tutorial: http://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html

3

Hi! Thanks for your reply. I know I can write my own custom function. However, I’m looking for a solution that directly overrides the existing implementation of torch.svd - for educational purposes mainly and because I’m wondering if it may be quicker if I can call LAPACK directly for example.

4

you cannot override torch.svd with your own function in a simple way.

5

Hello,

I encountered the same problem you described and could not find an answer.

would you be able to inform me what you did in the end?

Did you write a new function, and if so would you be able to share it?

1 Like
6

Hello Daniel,

yes I did not change any of the backend files in C. I just used the custom interface of PyTorch to define a new function and implemented backward in it myself using python functions. it worked okay i think. I can probably dig it out for you later today. What linear algebra operation do you require? svd?

7

That would be fantastic. Thanks!

Yes I require SVD

In fact, I am also searching for implementation of svds (only k largest
values of SVD)

8

FYI, svd is now availble in PyTorch master with full Variable support (forward + backward).

3 Likes
9

Does Simon’s comment help you or do you still require help? Wasn’t aware that this functionality got added already, but it’s good to know! Thanks Simon!

1 Like
10

Simon’s comment helped me out

But thanks for the offer!

11

Hi Simon,

I updated to torch 0.3 but am still getting error "the derivative for ‘svd’ is not implemented"
when using “loss.backward()”

am I doing something wrong?

12

Daniel, the derivative of svd is implemented after we released 0.3.
You can wait for the next release, or install from source using instructions at: https://github.com/pytorch/pytorch#from-source

1 Like
13

When you say install from source
is there a source to install from?
(I ask since you said svd is implemented)

14

Maybe I don’t understand your question, but you should use soumith’s link and follow the install guide.

15

apologies.
I am new to Torch

I just understood the previous comment

thanks for the help!

1 Like