Hello everyone,
I am currently testing the influence from soft error on the MobileNetV3 large. I am basically injecting bitflips into the model weights after the training has been completed, afterwards I am testing the impacts of the soft errors with the test dataset, whether there has been a drop in accuracy or not.
My result shows that especially the MSB can have an large effect for float32 model weights since values can change from 0.0xx to 1*e^x for example.
I want to prevent the model from basically computing the output with such large weights. And I came across the troch.clamp function.
From my understanding torch.clamp sets a min and max range for a tensor. So would it be possible to insert torch.clamp to basically mitigate the effect of large numbers after the training actually has completed since I don’t want to clamp the weights during the training. I want to prevent the model to “use” these large weights which completely fall out of the scale of the rest.
As a side note, i am using the already implemented MobileNetV3_large model from torchvision. So maybe there is a way to implement the range restriction without actually changing the models architecture.
For reference I am thinking of something similar to this GitHub - DependableSystemsLab/Ranger
Any help or suggestions would be appreciated.