Hi,
Could anyone help me?
After several batch training in every epoch, I tested my network on the validation set. Here is my pseudocode:
for i in range(epoch):
####### train #############
for j , batch in enumerate(train_loader):
pred = net(batch)
loss = loss_function(pred, label)
loss.backward()
optimizer.zero_grad()
optimizer.step()
###### validation ##########
if j % 10 ==0:
net.eval()
for jj , batch2 in enumerate(validation_loader):
pred2 = net(batch2)
loss = loss_function(pred2, label)
After the validation, my network parameter‘’s (e.g. nn.Parameter(torch.ones(1))) gradient becomes 0!!!
But I rewrite my pseudocode as follows:
for i in range(epoch):
####### train #############
for j , batch in enumerate(train_loader):
pred = net(batch)
loss = loss_function(pred, label)
loss.backward()
optimizer.zero_grad()
optimizer.step()
###### validation ##########
if j % 10 ==0:
net.eval()
for jj , batch2 in enumerate(validation_loader):
pred2 = net(batch2)
loss = loss_function(pred2, label)
**net.train()**
The only difference is that I add “net.train()” after the validation, then the gradient of nn.Parameter would not become 0.
So my question:
I should add this “net.train()” after the validation everytime, or I just did something wrong?