Enforcing zeros values in CNN weights

I’m trying to learn only part of the kernel and enforce the other values to be zeros. Do you have any idea how to do it? I would like the net to learn only the nonzero values and to remain the other as is.
Calling to nn.Parameter over all kernel will cause to learning also the values that I don’t want to learn.
Also. I can’t declare that part of the kernel will requires grad and the others not.
Thanks in advance!