How to create a new loss function on weights of fc.weight

1

Hello, I want to define a new loss function. Specifically, with the L1 norm applied on the weight of FC layer, my aim is to learn sparse feature. The code is following:

    params = model.state_dict()
    p = params['module.fc.weight']
    loss  = nn.CrossEntropyLoss()(output, target_var) + lambda*L1Loss(p)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

here is my loss function

def L1loss(weight)
   loss = torch,norm(weight, p=1)
   return loss

Obviously, it does work. The L1 loss dosen’t decrease, but the CrossEntropyLoss seems well.

So, How to combine them? or my definition on L1 loss is wrong?

1 Like
2

did you try to increase the lambda parameter for L1Loss? I cant think of any other reason why it wont decrease

3

Is it feasible to add L1 norm to the loss?