Different masks applied to conv filters at different pixels

[ptrblck]

I think the easiest method would be to apply a manual convolution by unfolding the input and apply the filter to each patch manually.
Here is a small code example.