Implementing the adjoint operator of a convolutional layer

Hello everybody,

that’s my first post, so please be kind!

I am currently trying to implement the adjoint operation of a convolutional layer (in 2D and 3D).
More precisely, let c be a convoltutional layer with a pre-defined kernel, e.g. a 3x3 kernel.
What is the easiest and most efficient way of implementing the adjoint operation of c, i.e. c*?
Note that I am NOT looking for a transposed convolution, but I really need the adoint operator of c, i.e. it has to fulfill
<cx,y> = <x,c*y> for any x and y.

Thank you very much in advance for any help!
Cheers,
Andreas

I’m not sure, if you mean the Hermitian adjoint, but if so I assume you would like to inverse the convolution?
If so, then note that inverse filtering is often not a good idea, if it doesn’t account for the additional error signal in the input:

(f * g) + e = h

If you know the type of noise (e.g. Gaussian) you could use a Wiener deconvolution.

For a general use case you could use the FFT and reverse the (noise free) convolution via:

F = H / G

and apply the inverse FFT again.

exactly, i need the hermitian adjoint of the convolution-operation c, say c^H The reason is that I am considering a functional F(x) = 1/2*||cx -y||_2^2, where c is just a simple conv.layer with no activation function. Considering the derivative of F with respect to x I get c^H(cx -y). Thus, I need the adjoint operator of c. Since a conv. can in general be associated with a linear operator (i.e. a matrix), if I had the matrix as it was, it would be easy, cause I would just use A^H. But I would like to be able to express the operation A^H by means of the same function which I am using in the implementation, i.e conv2d or conv3d. I guess using FFT in that case does not help, since in the end I end up with c^H(cx -y) anyway.
But thanks again :slight_smile:

Hello Andreas!

ConvTranspose2d should do what you want (at least in the plain-vanilla,
2d case – I didn’t check 3d or various stride / padding combinations).

Here is an example script:

import torch
torch.__version__
torch.random.manual_seed (2020)

cnva = torch.nn.Conv2d (1, 1, 3, padding = 1, bias = False)
cnvt = torch.nn.ConvTranspose2d (1, 1, 3, padding = 1, bias = False)
with torch.no_grad():
    _ = cnvt.weight.copy_ (cnva.weight)

s = torch.randn ((1, 1, 7, 7))
t = torch.randn ((1, 1, 7, 7))
(cnva (s) * t).sum()
(s * cnvt (t)).sum()

Note the use of the same weight matrix for ConvTranspose2d as for
Conv2d.

And here is its output:

>>> import torch
>>> torch.__version__
'1.6.0'
>>> torch.random.manual_seed (2020)
<torch._C.Generator object at 0x7efc1f81c6f0>
>>>
>>> cnva = torch.nn.Conv2d (1, 1, 3, padding = 1, bias = False)
>>> cnvt = torch.nn.ConvTranspose2d (1, 1, 3, padding = 1, bias = False)
>>> with torch.no_grad():
...     _ = cnvt.weight.copy_ (cnva.weight)
...
>>> s = torch.randn ((1, 1, 7, 7))
>>> t = torch.randn ((1, 1, 7, 7))
>>> (cnva (s) * t).sum()
tensor(3.5478, grad_fn=<SumBackward0>)
>>> (s * cnvt (t)).sum()
tensor(3.5478, grad_fn=<SumBackward0>)

To understand the adjoint, think of a 2d 100x100 image as being a
vector of length 1,000,000. The 2d convolution can be represented
(inefficiently) as a 1,000,000 x 1,000,000 matrix with a particular
banded structure. Because it’s real, its adjoint is just its transpose.

ConvTranspose2d represents (algorithmically) the appropriate
banded structure for that transposed 1,000,000 x 1,000,000 matrix.

(If pytorch were to support convolutions with complex weights, you
would presumably use the complex conjugate of Conv2d’s weights
for ConvTranspose2d.)

Good luck.

K. Frank

1 Like

Hello,
perfect, this looks like what I was looking for. Exactly, I am aware of the interpretation of the conv as an application of a matrix to a vector. I knew that for 2D, one typically only has to rotate the filter to obtain the tranposed operation. But since for 3D there are multiple possibilities to rotate the filter, I was wondering if there was a function which does this without me having to take care of how exactly to manipulate the filter depending on the dimensionality of the problem.
Thanks a lot again! :slight_smile:

Hello Andreas!

Did you try this scheme with Conv3d? (I didn’t.) Did it work?

Best.

K. Frank

Hi Frank,
yes, it worked fo 3D as well. :slight_smile: Thanks again!

Best,

Andreas