Hi everyone.

**Here is my problem :**

I’ve been playing around with some neural networks lately, trying to compare different kinds of normalization layers on a simple problem (namely VGG16 on the CIFAR10 dataset).

However the loss graph for only one model is really noisy, so I decided to plot a mean of the losses for 5 runs of the same architecture. Let L_it be the loss for the i-th model on the minibatch at time t, each of my point p_t is computed by the following :

I’m also plotting a rolling loss to smooth everything a bit, computed as follows :

And here is the kind of result that I get :

The steps which are appearing between every epochs shouldn’t be there I guess… I talked to my teachers about it, they all agree that it isn’t normal, but none of them has been able to find the problem.

**Concerning the code I use to produce those results :**

I really tried to keep it as simple as possible, and it follows really closely the code provided in the Pytorch tutorial for CIFAR10.

My DataLoader is created with this code :

```
train_transform = transforms.Compose([transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize((.5, .5, .5), (.5, .5, .5))])
trainset = torchvision.datasets.CIFAR10(root = "../data_CIFAR10", train = True, download = False, transform = train_transform)
trainLoader = torch.utils.data.DataLoader(trainset, batch_size = bs, shuffle = True, drop_last = True)
```

My model, the criterion and the optimizer are here :

```
model = models.vgg16_bn(num_classes = 10)
model.to(gpu)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr = 0.003)
train_losses = train_model(model, trainLoader, criterion, optimizer, nb_epochs, gpu)
```

And finally, here is the `train_model function :

```
def train_model(model, trainLoader, criterion, optimizer, epochs, gpu):
train_losses = []
for epoch in range(epochs):
for i, data in enumerate(trainLoader, 0):
inputs, labels = data[0].to(gpu), data[1].to(gpu)
optimizer.zero_grad()
outputs = model(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
train_losses.append(loss.item())
if(i%100==99):
print("[%d, %5d] loss : %.5f" % (epoch + 1, i + 1, loss.item()))
print("Finished training\n\n")
return train_losses
```

Did I miss anything obvious? I tried to describe my problem as precisely as possible (while keeping it simple), but if you need more information please ask.

Thanks.

(PS : what’s the best way to integrate LaTeX equations in a post?)