Implementing custom layer for cnn

1

I want to write a custom layer [Normalized Correlation Layer] in my cnn architecture. Where should I write this new method. As far as I know this layer is not implemented in pytorch.

2 Likes
2

You need to write a custom class which inherits from nn.Module and override the forward() function.
http://pytorch.org/docs/notes/extending.html#extending-torch-nn

Example : http://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-custom-nn-modules

5 Likes
3

Thank you for the help! I will have a look at the references.

4

@pvskand did you get it to work? can you share it?

5

@deepcode are you talking about the NCC layer or implementing a custom layer in general?
I was able to write the code adn get it to work. If you want to write a custom layer, @vinayakvivek’s answer is very apt [override the forward() function].

If you are talking about the implementation of NCC layer, well apart from the NCC, I also did some major changes in the layer to suite my architecture so I guess it won’t be of much use to you. But still if you want to have a look at the code, do let me know!

6

@pvskand I would like to see the implemetnation of the NCC layer. I am trying to implement my own layer and want to see some examples to work with. Thank you!

7

Hey @deepcode, here you go.
Implementation of NCC layer
Please note that I have done very few unit tests so there might be some error! Do let me know if there is any.

1 Like
8

Given the class implementation of a layer,what is the syntax to build a neural network which will have this layer as one of its many layers?

Going by the TwoLayerNet example, the model is created by a statement as follows:

model = TwoLayerNet(D_in, H, D_out)

This approach is not helpful in my more general setting.

When I look at the PyTorch example for MNIST, I see the following:

...
import torch.nn as nn
import torch.nn.functional as F
...
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1)
        self.conv2 = nn.Conv2d(20, 50, 5, 1)
        self.fc1 = nn.Linear(4*4*50, 500)
        self.fc2 = nn.Linear(500, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2, 2)
        x = x.view(-1, 4*4*50)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

Will my layer be called in the manner F.relu is called?

9

If your layer is a pure functional method, you could simply define it as a python function via def and call it in your forward method of the model.
On the other hand, if your layer contains parameters or buffers, it might be simpler to create a class deriving from nn.Module (same as your Net definition).

1 Like
10

Thank you. Right now, my layers are functional in nature. No parameters. I will work further on this and see.

12

I have been implementing layers for my application. One of my layers has an explicit backward function. In that layer, the forward function takes a tensor input. However, it creates a numpy copy of it and uses it. This is because inside this layer there is a call to another library which computes the output of this layer. The forward function assigns that library’s output to a tensor and outputs that tensor. A loss function is computed based on this output. I can see that this is breaking the backpropagation path. I get the error: “element 0 of tensors does not require grad and does not have a grad_fn”. But how to fix it? How to make PyTorch bridge the path from this Layer’s output to the previous layers? I have tried setting requires_grad = True for the output tensor, inside the forward function, for which I get the error “leaf variable has been moved into the graph interior”. I have tried setting requires_grad = True, outside, i.e., in the main function after this layer is called and before the loss function. I get the error “unsupported operand type(s) for *: ‘float’ and ‘NoneType’”, when I update my parameters through gradient descent.

13

Have you followed the steps in this docs?
Could you post the custom method you are using, so that we could have a look? A simplified version would also be OK. Somehow the computation graph might be broken, and I assume the model works without this custom layer?

14

Thanks a lot for your reply. Please, see below a toy code, with a toy layer. This toy layer treats the input in the same manner as my custom layer: creating a numpy array using copies of values from the input. This toy layer has an explicit but simple backward function, which is the case for my custom layer, also. Thus, this code behaves in the same manner as my original code.

So far, I have been testing optimization problems only. Hence, this toy example is also an optimization problem. I can define optimization problems without that custom layer, also and they behave as expected. So, there is a problem with that custom layer.

Yes, I had come across the link that you have shared earlier.

import torch
import numpy as np

class elemProdLayer(torch.nn.Module):

def __init__(self):
    super(elemProdLayer, self).__init__()

def forward(self, scaleVec, oneVec):

    self.oneVec = oneVec

    vecSiz = len(scaleVec)

    scaleVecLocal = torch.zeros(vecSiz)

    for iVec in range(vecSiz):
        scaleVecLocal[iVec] = scaleVec[iVec].item() 

    return torch.mul(scaleVecLocal, ipVec)

def backward(self, opVec):
    return torch.mul(self.oneVec, opVec)

device = torch.device(‘cpu’)

N = 5

oneVec = torch.ones(N, requires_grad = False)

ipVec = torch.randn(N, device=device, requires_grad = True)

tgtVec = torch.randn(N, device=device, requires_grad = False)

stepSiz = 1e-2

myElemProdLayer = elemProdLayer()

for t in range(500):
scaleVec = myElemProdLayer(ipVec, oneVec)

scaleVec = torch.tensor(scaleVec, requires_grad = True)

loss = torch.nn.MSELoss()(tgtVec, scaleVec)

loss.backward()

with torch.no_grad():

  ipVec -= stepSiz*ipVec.grad

  ipVec.grad.zero_()
1 Like