Locally connected layers

#1

Are there any plans to provide optimized locally connected layers in Pytorch, like LocallyConnected1D/LocallyConnected2D in Keras (convolution without weight sharing)?

https://keras.io/layers/local/

8 Likes
#2

As far as I understand this layer each window position will have its own set of weights.
So if we are using an input of shape [1, 3, 24, 24] and out_channels=2, kernel_size=3, stride=1 without padding, we will have 3*3*3*2*(22*22) = 26136 weights. Is this correct?
If that’s the case, if created a quick and dirty implementation of this layer using unfold and a simple multiplication.
Note that I haven’t tested properly all edge cases, so feel free to report any issues you get with this implementation.
I don’t know, if we can somehow skip the computation of the spatial output shape, as I need it for the internal parameters.
Also, I guess the implementation is not the fastest one, but should be a starter:

from torch.nn.modules.utils import _pair

class LocallyConnected2d(nn.Module):
    def __init__(self, in_channels, out_channels, output_size, kernel_size, stride, bias=False):
        super(LocallyConnected2d, self).__init__()
        output_size = _pair(output_size)
        self.weight = nn.Parameter(
            torch.randn(1, out_channels, in_channels, output_size[0], output_size[1], kernel_size**2)
        )
        if bias:
            self.bias = nn.Parameter(
                torch.randn(1, out_channels, output_size[0], output_size[1])
            )
        else:
            self.register_parameter('bias', None)
        self.kernel_size = _pair(kernel_size)
        self.stride = _pair(stride)
        
    def forward(self, x):
        _, c, h, w = x.size()
        kh, kw = self.kernel_size
        dh, dw = self.stride
        x = x.unfold(2, kh, dh).unfold(3, kw, dw)
        x = x.contiguous().view(*x.size()[:-2], -1)
        # Sum in in_channel and kernel_size dims
        out = (x.unsqueeze(1) * self.weight).sum([2, -1])
        if self.bias is not None:
            out += self.bias
        return out


batch_size = 5
in_channels = 3
h, w = 24, 24
x = torch.randn(batch_size, in_channels, h, w)

out_channels = 2
output_size = 22
kernel_size = 3
stride = 1
conv = LocallyConnected2d(
    in_channels, out_channels, output_size, kernel_size, stride, bias=True)

out = conv(x)

out.mean().backward()
print(conv.weight.grad)
6 Likes
#3

Have you gotten around testing this? Looking to use it

#4

any plan to make it official ?

#5

If enough people are using this layer, we could think about writing an official (and most likely more performant) version of it. Do you have any papers on this particular layer or other references, since I’ve just used the Keras docs to replicate it.

7 Likes
#6

I guess this paper from facebook should provide enough motivation to implement LC layers.

It seems deepface (ensembles) is still the best one
https://cmusatyalab.github.io/openface/models-and-accuracies/

6 Likes
#7

@ptrblck
Is there any update on official release?

3 Likes
#8

Any update on this??

#9

Any update ? I really want an official one with sufficient tests

#11

I, too, want to use this. Any updates?

#12

No updates, but PRs are welcome. :slight_smile:
Would anyone of you be interested in working on it? CC @ming_li

#13

@ptrblck
It seems there already an PR since 2017, and seems many people needs this function, but that PR is still under review, can you take a look at it and what is the problem there? Thank you very much:)

#14

It seems the PR was abandoned and would most likely need a rewrite as the backend methods could have changed, so feel free to post your interest in the PR.

#17

Hello!
Are there any plans in the near future to make it official?