Model Optimization using Low Rank Transformation in MMDetection

Hello, I am new to mmdetection and model optimization. I am using mmdetection to use mask RCNN with SWIN transformer as the backbone. As the model is big I was trying to reduce its size.As far as I know , pruning as implemented in pytorch would not change the model size but would simply create a mask.I have also tried L1 unstructured quantization with success and was currently trying low rank transformations. I can see the transformation happening when I compare the print(model) before and after the low-rank transformation. However, the model is not able to infer anything. I see no bounding box that appear. The following is what I have tried :

# Setup a checkpoint file to load
checkpoint = 'checkpoints/mask_rcnn_swin-t-p4-w7_fpn_1x_coco_20210902_120937-9d6b7cfa.pth'
# initialize the detector
model = init_detector(config,checkpoint)

Once the model is initialized I use the following for low-rank transformation

def low_rank_transform(model):
    for module_name, module in model.named_children():
      if module_name=='proj':
      if len(list(module.children())) > 0:
      if isinstance(module, torch.nn.Linear):
        # print(W)
        # U,S,V=torch.svd(W)
        U, S, Vh = torch.linalg.svd(W)
        V = Vh.transpose(-2, -1).conj()
        W1 = U[:,:L]
        W2 = torch.diag(S[:L]) @ V[:,:L].t()
        if module.bias is not None:
          # print("W1:",W1)
          layer_1 = nn.Linear(in_features=module.in_features,
                            out_features=L, bias=True)
 = W2

          layer_2 = nn.Linear(in_features=L,
                            out_features=module.out_features , bias=True)
 = W1
          # print("Layer2",
          # break

          layer_1 = nn.Linear(in_features=module.in_features,
                            out_features=L, bias=False)
 = W2

          layer_2 = nn.Linear(in_features=L,
                            out_features=module.out_features , bias=False)
 = W1
          # =

          # print("we are now trying to replace",module,"with",svd_classifier_layers,"in a sequnetial way")
          # n =int(module_name)
          # model[n]=nn.Sequential(*svd_classifier_layers)
          setattr(model, module_name, nn.Sequential(*svd_classifier_layers))
          setattr(model, module_name, nn.Sequential(*svd_classifier_layers))

Then I simply call the above function by passing only the backbone of the model.

I can see the expected change in the architecture where a single Linear layer gets replaced by two linear layers. However, nothing happens during inference.

Can someone help me with what is that I might be doing wrong here?

I have observed on using the inference_detector() that the confidence of the model is relatively low and which is obvious. Can someone suggest a better way to do low-rank transformations on a model? or perhaps how to change the model’s score threshold so as to get bounding boxes in the image.