How to implement a kernel-wise convolution

[dongtuu]

OK, I find the answer I need. As I cannot delete this topic, I just paste the answer here.

Convolution that takes a function as kernel

Hooo sorry I misread your question, I though you wanted the weights to be an MLP. You want for every patch to have an MLP to generates the weights, and then apply these weights to this patch? In that case you will need to use unfold. From the example in the doc, you will need to generate w from inp_unf which contains every patch (L such patches). And since you want one weight per patch, your weights will be (N, patch_size, L, chan_out). Then replace the matmul that does the conv by an element …