How to create a sparse autoencoder neural network with pytorch

1

how to create a sparse autoEncoder neural network with pytorch,tanks!

2

in a sparse autoencoder, you just have an L1 sparsitiy penalty on the intermediate activations.

You can create a L1Penalty autograd function that achieves this.

import torch
from torch.autograd import Function

class L1Penalty(Function):

    @staticmethod
    def forward(ctx, input, l1weight):
        ctx.save_for_backward(input)
        ctx.l1weight = l1weight
        return input

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_variables
        grad_input = input.clone().sign().mul(self.l1weight)
        grad_input += grad_output
        return grad_input

Then you can use it like this:

import torch.nn.functional as F

class SparseAutoEncoder(torch.nn.Module):
  def __init__(self, feature_size, hidden_size, l1weight):
    super(SparseAutoEncoder, self).__init__()
    self.lin_encoder = nn.Linear(feature_size, hidden_size)
    self.lin_decoder = nn.Linear(hidden_size, feature_size)
    self.feature_size = feature_size
    self.hidden_size = hidden_size
    self.l1weight = l1weight
    
  def forward(input):
    # encoder
    x = input.view(-1, self.feature_size)
    x = self.lin_encoder(x)
    x = F.relu(x)

    # sparsity penalty
    x = L1Penalty.apply(x, self.l1weight)
    
    # decoder
    x = self.lin_decoder(x)
    x = F.sigmoid(x)
    return x.view_as(input)

I didn’t test the code for exact correctness, but hopefully you get an idea.

References:

  1. https://github.com/Kaixhin/Autoencoders/blob/master/models/SparseAE.lua
  2. https://github.com/torch/nn/blob/master/L1Penalty.lua
6 Likes
3

@smth

Just can’t connect the code with the document.
http://deeplearning.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity

Can you show me some more details? Where is the parameter of sparsity? What is l1weight? Is it the parameter of sparsity, e.g. 5%? What is the loss function?
Is there any completed code?

Thanks in advance!

1 Like
4

Why put L1Penalty into a Layer? Why dont add it to the loss function?
In another words, L1Penalty in just one activation layer will be automatically added into the final loss function by pytorch itself?

1 Like
5

what is the difference with adding l1 or KL-loss to final loss function ?

6

This code doesnt run in Pytorch 1.1.0! I keep getting the backward() needs to return two values not 1!
Edit :
You need to return None for any arguments that you do not need the gradients. so the L1Penalty would be :

import torch
from torch.autograd import Function

class L1Penalty(Function):

    @staticmethod
    def forward(ctx, input, l1weight):
        ctx.save_for_backward(input)
        ctx.l1weight = l1weight
        return input

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_variables
        grad_input = input.clone().sign().mul(self.l1weight)
        grad_input += grad_output
        return grad_input, None
7

Did you find your answer?