DeepLabV3+ Transfer Learning

Does anybody has a good understanding of the deeplabv3+ module or has some experience when applying transfer learning on semantic segmentation tasks which are based on the deeplabv3+ model?

The deeplabv3+ module consist of three main modules: backbone, ASPP and decoder

Does it make sense to freeze layers of the ASPP or decoder module during training? Or is it advisable to only freeze layers/layergroups of the backbone (e.g. resnet101)?