Sector Vector Machine Classifier

1

Hi. I have to implement an autoencoder network with a SVM as classifier. The input of the classifier is the latent space generated by the autoencoder. Also, I have to sum the loss of the AE and that of the SVM. I used this code:

    for epoch in range(NUM_EPOCHS):
    for batch_tr, (XTrain, YTrain) in enumerate(train_loader):
        XTrain = XTrain.to(device)
        optimizer.zero_grad()
        recon_batch, mu, logvar, latent_space = model(XTrain)
        predictions = svm_model(latent_space)
        loss_svm_train = svm_loss_criteria(predictions, YTrain)
        loss = loss_mse(recon_batch, XTrain, mu, logvar) + loss_svm_train
        svm_optimizer.zero_grad()
        svm_optimizer.step()
        loss.backward()

It generates the following error: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [128, 2]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. How can I fix it?

2

Could you post a minimal, executable code snippet to reproduce the issue, please?

3

This is what I have for the svm:

class SVMLoss(nn.Module):
    def __init__(self):
        super(SVMLoss, self).__init__()

   def forward(self, predictions, targets):
        return torch.sum(torch.clamp(1 - predictions.t()*targets, min=0))/TRAIN_BATCH_SIZE

Then, inside the main I used this:

optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE, 
weight_decay=L2_REGULARIZATION)
loss_mse = CustomLoss(1.5)
train_loss = []
svm_model = nn.Linear(LATENT_FEATURES, NUM_CLASSES)
svm_loss_criteria = SVMLoss()
svm_optimizer = torch.optim.SGD(svm_model.parameters(), lr=LEARNING_RATE, 
momentum=MOMENTUM)

I think the problem is to do the gradient backpropagation of two different classes.

4

Your code is unfortunately not executable, but the posted code snippets work fine:

class SVMLoss(nn.Module):
    def __init__(self):
        super(SVMLoss, self).__init__()

    def forward(self, predictions, targets):
        return torch.sum(torch.clamp(1 - predictions.t()*targets, min=0))/TRAIN_BATCH_SIZE
    
TRAIN_BATCH_SIZE = 2
LEARNING_RATE = 1e-3
L2_REGULARIZATION = 0.1
LATENT_FEATURES = 2
NUM_CLASSES = 2

model = nn.Linear(LATENT_FEATURES, NUM_CLASSES)
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE, weight_decay=L2_REGULARIZATION)
loss_mse = nn.MSELoss()
svm_loss_criteria = SVMLoss()

x = torch.randn(TRAIN_BATCH_SIZE, LATENT_FEATURES)
y = torch.randn(TRAIN_BATCH_SIZE, NUM_CLASSES)

for epoch in range(1000):
    optimizer.zero_grad()
    out = model(x)
    loss = loss_mse(out, y) + svm_loss_criteria(out, y)
    loss.backward()
    optimizer.step()
    print("epoch {}, loss {}".format(epoch, loss.item()))
5

The code now works, thank you very much, but you removed the autoencoder optimizer:

optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE, 
weight_decay=L2_REGULARIZATION)

is there a way to mantain it together with the svm optimizer? Substantially, I’d like to train both the AE and the SVM.

6

Yes, I removed it as it was undefined in your code snippets, but it should be possible to use both optimizers with both models.

7

Sorry, I forgot the autoencoder model:

model = Autoencoder(1.5).to(device)

However, I solved by keeping only an optimizer for both the models:

params = list(model.parameters()) + list(svm_model.parameters())
optimizer = optim.Adam(params, lr=LEARNING_RATE, weight_decay=L2_REGULARIZATION)

Can be a good method?

8

Sure, using a single optimizer should also work assuming both models should be updated with Adam now.