Weird behavior when obtaining training accuracy

Your running_{variables} should be within the loop so they are refreshed each epoch.

The acc_train[epoch] should be outside the epoch loop.

Also look into this: Order of backward(), step() and zero_grad() for your ordering of backwards and zerograd.

Generally, it is more with the data or the hyper-parameters which cause weird trends like this and not a broken training logic. Is this happening every time or is this one off?