Obtaining Toeplitz matrix for convolution

1

In this issue @ezyang references an implementation of convolutions that uses the Toeplitz matrix.

I have a state_dict (and also a nn.Module class) from a network and explicitly need these Toeplitz matrices for further calculations but I admittedly have not a strong grasp on the things going on in ATen and how I could use that directly in Python. Is there a way to do this?

2

Hi McLawrence,
Please refer to the nn.Unfold module. This would enable you to generate the toeplitz matrices(column matrix).
Hope this helps!

1 Like
3

@Mazhar_Shaikh Using unfold, I can create a matrix from the input and do a matrix multiplication with the kernel vector.
However, what I need would be a matrix from the kernel, not the input. Which seems not possible with unfold (as the kernel is smaller than the input). Do you know of an alternative?

4 Likes
4

I’m not sure if this could be useful. However, since I recently needed a Toeplitz matrix in custom layer and this was the only resource I could find, I prepared a function that returns the Toeplitz matrix for real values using the Unfold module.

Here’s the function:

    def pytorch_toeplitz(V):
    '''
    It creates the Toeplitz matrix for each row in V.

    INPUT:
    V: torch tensor (batch_size x d)

    OUTPUT:
    T: (batch_size x d x d)

    EXAMPLE:
    V = torch.tensor([[1, 0.5, 0.1], [1.3, 0.9, -0.1]])
    T = pytorch_toeplitz(V)
    print(T.shape)
    print(T[0])
    print(T[1])
    '''
        
    d = V.shape[1]
    A = V.unsqueeze(1).unsqueeze(2)
    A_nofirst_flipped = torch.flip(A[:, :, :, 1:], dims=[3]) 
    A_concat = torch.concatenate([A_nofirst_flipped, A], dim=3) 
    unfold = torch.nn.Unfold(kernel_size=(1, d))
    T = unfold(A_concat)
    T = torch.flip(T, dims=[2])

    return T