I am trying to train an auto-encoder decoder type of network where I have few set of convs and then a flatten/reshape layer to a single vector and again want to reconstruct the image back.

But does this handle the non linearity? In case I have my conv layer as follows: l1 = F.sigmoid(self.conv1(x)) l2 = F.sigmoid(self.conv2(l1)) ... fc = ...# have set of fc layers using nn.linear and then reconstuct them back by interchanging the input and output dimensions.

it does not handle the nonlinearity.
From the description of lasagne’s InverseLayer, it uses the derivative, so essantially, it effectively provides the backpropagation step of the layer it is based on. You would need to do this yourself (using d/dsigmoid(x) = sigmoid(x)*(1-sigmoid(x)), so reconstruct_2 = self.deconv2(fc*(1-fc)) or so) or use the torch.autograd.grad function.
My understanding is that for the backward, you would want the nonlinearity-induced term before the convolution.
This is, however, not necessarily something that reconstructs anything.