I am running the neural network for the train data for 10 iterations, with a learning rate of 0.01 and the accuracy increases about 20% after those 10 iterations.
I’ve used train/test in regression previously, and it’s mainly testing the data on a set of parameters that I computed during the testing/validation phase.
But for neural networks, the only hyperparameters are the learning rate and epochs/iterations.
How do I run the test data? Do I run it for the same number of epochs as the training or only one, and that becomes my testing accuracy?
It is common to split your data into training set and test set (where test set is much smaller than training set).
This way you can train on the training set and let the network learn from it while you test on the test set from which the network is not allowed to learn.
For how often you should test…
Well it’s really up to you. You could test once the whole training is finished or test once after every epoch to see how accuracy is changing over time.
If you test set is not too large I would recommend testing after every epoch.
You can test your network by running your whole (usually the whole if it is not too large) test set through it once.
The very very important difference to training is, that you do not use back propagation on the output you get from the test set! So no Tensor.backward() on the output of your test data.
In fact, it is best practice to wrap the testing code into with torch.no_grad():
with torch.no_grad():
#testing code
this disables gradient calculation inside it. Which makes sure you cannot call Tensor.backward() meaning you cannot learn from your test data and it can also reduce memory consumption.
Thank you for your reply, it’s a lot more clearer now. So just to confirm, I just run the test data for only one iteration of the forward pass with the same learning rate or do I also have to first extract the weights from the last iteration of the training phase?
and I think you mean epoch and not iteration over here
but I hope an example helps:
Let’s say I have 10,000 images. I split it into 8,000 images for my training set and 2,000 images for my test set (just an example). I then train my network for one epoch on the training set (with a batch size of 100 that would be 80 iterations). Here I use back-propagation to calculate the loss and give the loss to the optimizer (with the specified learning rate) to update the weight so the network learns.
After that I test the network by sending the test set through the network once, so one epoch (which would we 20 iterations with a batch size of 100) but here I do not use back-propagation! I do also not use the optimizer and I don’t even need a learning rate since I am not making any changes to the weights.
I just get the output and check it against the labels to calculate my accuracy.