How to implement asymmetric groups in Convolutional layer?

About the groups parameter in conv2d, how to get asymmetric groups through this parameters ?

As the figure below from Lecun 1998 , the output feature maps based on subset of input maps, could we implement this by groups ?

No, it’s not possible to do it with the groups option.
The only way I see of doing it would be to hackily wrap SpatialConvolutionMap (from Lua torch) in pytorch, but it doesn’t have GPU kernels implemented, so you’d need to run it on the CPU.
Furthermore, if I remember properly, it is unlikely that we will have optimized GPU kernels for SpatialConvolutionMap.