Parameter-Efficient Fine-Tuning for CNN

vision
1

Dear community,

  1. Is there some Parameter Efficient Fine-Tuning technique like LORA(low rank adaptation) is available for CNN to reduce the GPU memory usage while training/fine-tuning the network?

  2. Is it possible to apply LoRA for CNN kernels which will be of (3X3, 5X5, 7X7)?

Any article recommendations will be highly helpful. Thanks

2

It seem like Microsoft has written LoRAConv layer, you can find it here.

3

I implemeted a ton of PEFT methods for both CNNs and ViTs. Check my paper here - [2305.08252] Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity

4

Do you have your code somewhere? I’m particularly interested in trying SSF, but it performs super bad so I must be doing something wrong

5

The official code is here - GitHub - dongzelian/SSF: [NeurIPS'22] This is an official implementation for "Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning".

6

Thank you for your response! I read in your paper that you’ve added SSF to a ResNet50, did you add it to only the ResBlocks? And throughout the entire model? I tried to add it to the resblocks of a segresnet like
` scale, shift = init_ssf_scale_shift(in_channels)
self.ssf_scale = scale
self.ssf_shift = shift

def forward(self, x):
    
    if self.tuning_mode == "ssf":

        identity = x
        
        x = self.norm1(x)
        x = self.act(x)
        x = ssf_ada(self.conv1(x), self.ssf_scale, self.ssf_shift)

        x = self.norm2(x)
        x = self.act(x)
        x = ssf_ada(self.conv2(x), self.ssf_scale, self.ssf_shift)

        x += identity`.

Then I only train the params with ‘ssf’ in their name, and this gives very poor results.

7

Yes, LoRA can be adapted for CNNs to reduce GPU memory usage. Check “LoRA: Low-Rank Adaptation of Large Language Models” for foundational concepts.