Using two optimizers for the same model (1 full precision, 1 binarized)

Hello everyone,

I am currently working on a binarized optimizer for CNNs. Right now I am using a model which has both full precision and binarized weights (not in the same module).

What I am trying to do is something like this:

bin_params = []
fp_params = []

for name, param in model.named_parameters():
     if "bin" in name:
          bin_params.append(param)
     else:
          fp_params.append(param)

This for creating two different sets of parameters. Then, I define my optimizers

fp_optimizer = optim.SGD(fp_params, lr = lr, momentum = momentum, weight_decay = weight_decay)
bin_optimizer = BinaryOptimizer(binary_params, 1.0)

optimizer = MultipleOptimizer(fp_optimizer, bin_optimizer)

Where

class BinaryOptimizer(Optimizer): 

    def __init__(self, params, scale=required):     
        if scale is not required and scale <= 0.0:
            raise ValueError("Invalid scale value: {}".format(lr))
            
        defaults = dict(scale=scale)
        
        super(BinaryOptimizer, self).__init__(params, defaults)
        
        
    def __setstate__(self, state):
        super(BinaryOptimizer, self).__setstate__(state)
        
        
    def step(self, closure=None):
        loss = None
        if closure is not None:
            loss = closure()
            
        for group in self.param_groups:
            scale = group["scale"]
            
            for p in group["params"]:
                if p.grad is None:
                    continue
                    
                grad = p.grad
                
                flips = # Some value
                                
                p.mul_(flips)
                
        return loss

My problem is that when p.mul_(flips) runs, I have the following error:

RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

I don’t know where the problem is, do you see it?

Thank you