I'm getting 0% accuracy on every epoch while loss decreases

1

Hello everyone and thanks in advance for taking the time to read through my issue. I’m currently trying to tackle regression on a Time Series (Bitcoin prices in the last 5 years). To spare you all the boring details my dataset is as follows after preprocessing:

0       0.000846
1       0.001299
2       0.002159
3       0.001813
4       0.001773
          ...   
1822    0.680117
1823    0.689799
1824    0.687676
1825    0.690270
1826    0.715508
Name: Close, Length: 1827, dtype: float64

This is basically the Closing price for every day in the past 5 years (its normalized based on the entire dataset then extracted). From this, I want to create a “sliding window” dataset so I can use the previous 50 days to predict the price of the next day. My custom dataset is as follows:

class TimeSeriesDataset(Data.Dataset):   
    def __init__(self, data, window):
        self.data = torch.Tensor(data.values)
        self.window = window
    
    def __getitem__(self, index):
        return (self.data[index:index+self.window], self.data[index+self.window])
    
    def __len__(self):
        return self.data.__len__() - (self.window)

I’ve manually split the data into training and testing (80% to training) and my network architecture is this:

self.hidden1 = torch.nn.Linear(50, 25) # hidden layer
self.hidden2 = torch.nn.Linear(25, 25) # hidden layer
self.out = torch.nn.Linear(25, 1)     # output layer

and these are my training and testing methods:

def train(dataset, model, loss_fn, optimizer):
    size = len(dataset)
    model.train()
    for batch, (X, y) in enumerate(dataset):
        X, y = X.to(device), y.to(device)

        # Compute prediction error
        pred = model(X).squeeze(-1)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch
            print(f"loss: {loss:>7f}  Batch: [{current:>5d}/{size:>5d}]")

def test(dataset, model, loss_fn):
    size = len(dataset)
    num_batches = len(dataset) - 50
    model.eval()
    test_loss, correct = 0, 0
    with torch.no_grad():
        for X, y in dataset:
            X, y = X.to(device), y.to(device)
            pred = model(X).squeeze(-1)
            test_loss += loss_fn(pred, y).item()
            testing_acc = torch.sum(pred == y)
    test_loss /= num_batches
    print(f"Test Error: \n Accuracy: {testing_acc}%, Avg loss: {test_loss:>8f} \n")

If I now run this for 20 epochs I notice that on every epoch the loss drops really low, however my accuracy is always 0% because none of my predicted values match the labels.

I figured that I wouldnt be able to predict the price with good accuracy (because if I could then I could beat the market I guess ?) but 0% seems way too low (Or that something’s wrong) especially when loss is dropping. Am I calculating this the wrong way ? What can I try ?
Thanks again and sorry for the long post.

2

Hi Elemeo!

You are testing for the exact equality of floating-point numbers. In
general (except in cases of “special” values like 0.0) two floating-point
numbers, even if very nearly equal, are extremely unlikely to be exactly
equal. Therefore it is expected that you never have a “correct” prediction
and that your computed accuracy is 0% (even if your predictions are
actually very good).

As an aside, it is true that predicting tradable market prices is a difficult
problem. (To first approximation, the best prediction for today’s closing
price will be yesterday’s closing price.) If you think about it, many
challenging neural-network problems – say, distinguishing cats from dogs
or localizing all of the instances of people in an image – are altogether
quite easy for people to do. But ask yourself how easy it is for you to
predict today’s Bitcoin closing price (more accurately than just using
yesterday’s closing price) by looking at the time series up through
yesterday. That should suggest that this could be quite hard for a neural
network, as well.

Best.

K. Frank

3

I understand that absolutely and I don’t really expect anything amazing to come out of it, it’s more like an exercise and a way to get familiar with PyTorch.

Regarding floating points I figured that it would be almost impossible to get exact numbers with 2 decimals so accuracy will always be 0%.

Do you have any suggestions as to what I would use to measure accuracy ?

Thank you.

4

Hi Elemeo!

Just to be clear, in your above code you’re not checking whether the
floating-point numbers agree to 2 decimal digits of accuracy, but rather
whether they agree exactly, which is to say to about 7 decimal digits of
accuracy.

A very reasonable measure of accuracy could be the mean-squared-error
between your predictions and your targets. (This could also be a good
choice for the loss function you use for training.)

If your data are market prices given in, say, dollars, the units of the
mean-squared-error will be dollars-squared. So it might be more intuitive
to repackage the same information as the root-mean-squared-error.
Thus, using the variables from the code you posted:

(pred - y).mean().sqrt()

This will have units of dollars, which might be a little easier to think about.

Note, however, that your time series starts of very small and grows over
time. This means that errors in your predictions will likely be much smaller
and count much less early on rather than later. For this reason you may
prefer to use fractional errors, e.g., something like (pred - y) / y.

If it’s important that your use case involve predicting tradable market
prices, then bear in mind that is it is relatively easy, but uninteresting,
to predict the price itself – just use yesterday’s price as your prediction.
Much more interesting – and also much harder – is to predict the
change in price.

I would suggest that you try to predict fractional price changes (“returns”),
or, perhaps more elegantly, logarithmic returns:

r_t  =  log (p_t / p_(t - 1))

where p_t is the price at time t, and so on.

The mean-squared-error, ((r_pred - r_targ)**2).mean() would be
both a sensible loss function for training and accuracy metric for
evaluation.

If you want to get fancy, you could even use the results of a (differentiable)
profit-and-loss trading “simulation” as your training loss and/or evaluation
metric. That is, you take a “position” based on how large your predicted
return is. If your prediction is in the right direction, you make a “profit”
on that position, and if it’s wrong, you take a “loss.”

Best.

K. Frank

1 Like
5

Will try, thank you for the suggestions.