Is there any good way to implement CP decomposition?

1

Hi all, I’d like to implement CP decomposition to downsize the tensors. As far as I know, PyTorch does not support these factorization for more than 2D. (I know for 2D there are some factorization functions)

CP decomposition factorize I*J*K tensor (we call it X) into I*R, J*R, K*R tensors (U, V, W respectively). Naively, I use stochastic gradient descent to get U, V and W. When composing, you need to add R tensors Y_1, Y_2,...Y_R, where

Y_r = outer(outer(U[:,r], V[:,r]), W[:,r])

I implemented this naively as below,

def outer(t1, t2):
    h, w = t1.size()
    c = t2.size()[0]
    return t2.repeat(h, w, 1) * t1.unsqueeze(2).repeat(1, 1, c)

out = variable(torch.Tensor(I, J, K))
for r in range(R):
     out += outer(torch.ger(U[:, r], V[:, r]), W[:, r])

When X is small then it works, but when X is large OutofMemoryError is thrown because Y_r is I*J*K tensor.

I think there might be two way to avoid this error, first is get ΣY_r directly from U, V, W instead of composing tensors from vectors. The other is creating and adding outer(torch.ger(U[:, r], V[:, r]), W[:, r]) effectively. But still I don’t have any idea to improve.

Does anyone has good solutions to this? Thank you for advance.

2

You can use tensorly package and choose pytorch as backend. Gives same results if you were to implement it yourself and say optimize the norm with Adam.

3 Likes
3

Thank you. Yes, recently I found tensorly. It’s super great!

4

You can use einsum for this situation if you need more specially operation on tensor.
like this “torch.einsum(‘ir,jr,kr->ijk’, self.A, self.B, self.C)”