How to conduct convolution over part of a feature map?

I am gonna to conduct convolution over parts of a feature map instead of the whole feature map, which is somewhat like the operation in this paper[Image Inpainting for Irregular Holes Using Partial Convolutions(https://arxiv.org/abs/1804.07723).

The difference between what I want to do and the paper is that I want to save the conputational cost to speed up inference by abandon convolution operation in non-target loaction of a feature map.

Specifically, suppose there is a mask with the same size of a feature map , which is used to annotate wether to take this location as the center of a convolution operation. 1 denotes doing, 0 denotes not doing. By this mask, convolution operation can be conducted only at the location marked 1, For those location marked 0, just preserve the value at like before. This idea is to save computation cost.

Is it able to achieve this idea in pytorch?