How to stop weight-decay from affectnig PReLU?

[falmasri]

Don't Regularize Me, Bro

Thanks for the tips, guys. Your suggestions, and looking at the .step() method of an optimizer, provide some key insight. Here’s a compact example for reference purposes, in case anyone else has this need: model = nn.Sequential( nn.Conv2d(...), nn.PReLU(...), nn.Conv2d(...), nn.PReLU(...), nn.Conv2d(...), ) params = [ { 'params' : model[0].parameters() }, { 'params' : model[2].parameters() }, { 'params' : model[4].parameters() }, { 'params' : model[1].parameters(), 'weight_d…