Multiple forward passes

1

Is it possible that uncommenting the line

out_u, mid_u = self.net(Iun)

may affect the performance as the loss does not depend on this line?
In my understanding commenting / uncommenting this line should not affect the test accuracy based on the net. However, it does. What could be the reason?

self.net.train()
for batch_idx in range(n):
	Il, lbls, Iun = self.getNextBatch()

	# labeled
	outputs, mid = self.net(Il)
	loss = self.criterion(outputs, lbls)
	self.optimizer.zero_grad()
	loss.backward()
	self.optimizer.step()
	
	# unlabeled
	out_u, mid_u = self.net(Iun)

acc = testModel()

However, the following two codes (CODE 1 and CODE 2) give me the same test classification performances (i.e., acc1 = acc2).

# CODE 1
self.net.train()
for batch_idx in range(n):
	Il, lbls, Iun = self.getNextBatch()

	# labeled
	outputs, mid = self.net(Il)
	loss = self.criterion(outputs, lbls)
	self.optimizer.zero_grad()
	loss.backward()
	self.optimizer.step()

    # unlabeled
    # out_u, mid_u = self.net(Iun)

acc1 = testModel()

# CODE 2

for batch_idx in range(n):
	Il, lbls, Iun = self.getNextBatch()

	# labeled
    self.net.train()
	outputs, mid = self.net(Il)
	loss = self.criterion(outputs, lbls)
	self.optimizer.zero_grad()
	loss.backward()
	self.optimizer.step()

    self.net.eval()
    # unlabeled
    out_u, mid_u = self.net(Iun)

acc2 = testModel()
2

of course any extra calculations result in decreasing performance.

may be i misunderstood your question. what do you mean by “test performance”? also, what do you mean by “Am I doing something wrong somewhere else?”. your code without “# unlabaled” line is a proper abstract code for any typical training procedure. perhaps, what is the purpose of this “# unlabeled” line?

3

I have refined my question. Test performance = classification accuracy on the test set based on self.net.
Am I doing something wrong somewhere else? = May be some bugs in the other part of my code?
I need to calculate the loss based on the #unlabeled, however, I havent done that so far.

4

Sorry I found it. It is due to the batch norm, and it seems my Il and Iun are normalized differently.

5

good to hear you’ve found an answer
in addition, I want to mention, that computations on GPU are non-detirministic, unless you set some torch.cudnn options manually. thus, two runs of the similar code fragments may produce different results. Also, randomness affects the result, which occurs from BatchNorm in your case, which you’ve said you use.

1 Like