Help with svd approximation

1

Hi I am trying to implement “Exploiting linear structure for efficient evaluation” paper and it requires me to perform SVD on the weight tensor and find its rank 1 approximation and now these rank 1 filters perform the convolution operation.

I wrote a simple test suite to verify this and I am getting good speed ups. Here is the code which does that:

# doing passes
res1 = np.zeros((oH, pw))
out = np.zeros((oH, oW))
t = time.time()
for i in xrange(oH):
    hst, hend = i*stride, i*stride+HH
    patch = padded_x[hst:hend, :]
    res1[i] = np.sum(patch*u, axis=0)
for i in xrange(oW):
    wst, wend = i*stride, i*stride+WW
    patch = res1[:, wst:wend]
    out[:, i] = np.sum(patch*v, axis=1)
elapsed = time.time() - t
print "time taken to do this awesome operation is {}".format(elapsed)
print out

I wanted to know if there is a way I can do this using pytorch and get the speed improvements?

2

All of the operations used here are supported, but we don’t support backpropagation through SVD. I’d suggest implementing your own Function to do that. You can find a tutorial in the notes.

3

Hi Adam,
I am trying to implement this in pytorch now, but i am not getting the right performance.

If you would guide me in the right direction, I would be grateful.

import numpy as np
import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.nn.functional as nnf
import time

image = torch.randn(512, 512)
image = Variable(image.view(1,1,512, 512))

u = torch.Tensor([1,2,1])
v = torch.Tensor([-1, 0, 1])
w = Variable(torch.ger(u, v))
w = w.view(1,1,3,3)

u = Variable(u.view(1,1,3,1))
v = Variable(v.view(1,1,1,3))

t = time.time()
res1 = nnf.conv2d(image, u, stride=1, padding=1)
out = nnf.conv2d(res1, v, stride=1)
print "time taken is {}".format(time.time()-t)
print out

t = time.time()
out1 = nnf.conv2d(image, w, stride=1, padding=1)
print "time taken by normal is {}".format(time.time()-t)
print out1
4

I don’t think it’s that surprising that a single convolution is faster. For each of them, you have to preprocess the image in some way, most libraries are optimized for 3x3 filters, since they’re one of the most commonly used sizes, and you benefit from data locality if you only do a single conv (no need to transfer the image back and forth).

5

So i would not get any improvements if I implement the forward and backward pass for this as per your previous suggestion of extending “autograd”

Thanks

6

Correct. I just wanted to tell you how could you do that.

1 Like