Developing a demo classifier

I’m writing a demo code to predict the 2-class label for a dataset of 10-D inputs. Below, function _data generates the data:

import math
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

def _data(dimension, num_examples):
    # Create a simulated 10-dimensional training dataset consisting of 1000 labeled
    # examples, of which 800 are labeled correctly and 200 are mislabeled.
    num_mislabeled_examples = 20
    # We will constrain the recall to be at least 90%.
    recall_lower_bound = 0.9

    # Create random "ground truth" parameters for a linear model.
    ground_truth_weights = np.random.normal(size=dimension) / math.sqrt(dimension)
    ground_truth_threshold = 0

    # Generate a random set of features for each example.
    features = np.random.normal(size=(num_examples, dimension)).astype(
        np.float32) / math.sqrt(dimension)
    # Compute the labels from these features given the ground truth linear model.
    labels = (np.matmul(features, ground_truth_weights) >
              ground_truth_threshold).astype(np.float32)
    # Add noise by randomly flipping num_mislabeled_examples labels.
    mislabeled_indices = np.random.choice(
        num_examples, num_mislabeled_examples, replace=False)
    labels[mislabeled_indices] = 1 - labels[mislabeled_indices]

    return torch.tensor(labels), torch.tensor(features)

The code below shows my attempt, where predictor is the model and the loss function is chosen to be Hinge loss.

import math
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

dim = 10
N = 100
target, features = _data(dim, N)

class predictor(nn.Module):
    def __init__(self):
        super(predictor, self).__init__()
        self.f_1 = nn.Linear(10, 1)

    def forward(self, features):
        return self.f_1(features)

model = predictor()

optimizer = optim.Adam(model.parameters(), lr=1e-2)
loss = torch.nn.HingeEmbeddingLoss(margin=1.0, size_average=None, reduce=None, reduction='mean')

running_loss = 0
for _ in range(1000):
    optimizer.zero_grad()
    output = model(features)
    objective = loss(output, target)
    objective.backward()
    running_loss += objective.item()
    optimizer.step()

    print(running_loss)

My questions:

  1. I see my loss increase from zero to 20 and then dives deep into negative realm. I was wondering if my implementation is correct.
  2. I was trying to implement my predictor without using nn.Linear by defining the computations myself as:
class predictor(nn.Module):
    def __init__(self):
        super(predictor, self).__init__()
        self.weights = torch.zeros(dim, dim, requires_grad=True)
        self.threshold = torch.zeros(1, 1, requires_grad=True)

    def forward(self, features):
        return torch.matmul(self.weights, features) - self.threshold

but in the optimization process,

model = predictor()
optimizer = optim.Adam(model.parameters(), lr=1e-3) 

I get the following error:

ValueError: optimizer got an empty parameter list

I would appreciate direction or comments on how to fix these issues. Thanks.

self.weights = torch.nn.Parameter(torch.zeros(dim, dim, requires_grad=True))
self.threshold = torch.nn.Parameter(torch.zeros(1, 1, requires_grad=True))

should do.

1 Like

Notice that you are doing running_loss += objective.item() which is essentially adding up the loss from the previous epochs as well. Every iteration (in your code) constitutes an epoch because you are sending in all the features at once. Does it make sense?

Something like:

running_loss = []
for _ in range(1000):
    ...
    running_loss.append(objective.item())
    ...
print(running_loss)

should do.

1 Like

So without adding up, my objective.item() starts from around 0.5 and still goes negative. I was expecting the loss not to fall below zero (Although a self implemented version of this loss function works just fine).

Could it be because target has 0 s and 1 s in it unlike what the documentation suggests, that it should contain 1 s and -1 s. Is the behavior when you add this line target[target == 0] = -1 along expected lines?