Compute the batch size for combined losses?

autograd
1

Hello all, I have two losses: loss1 and loss2. The loss1 uses a batch size of 16, and the loss2 also uses a batch size of 16. What is the batch size of combined loss1+loss2? This is my code

trainloader1 = dataloader.DataLoader(data1, batch_size=16, shuffle=True)
trainloader2 = dataloader.DataLoader(data2, batch_size=16, shuffle=True)
trainloader1_iter = enumerate(trainloader1)
trainloader2_iter = enumerate(trainloader2)
for iters in range (1000):
	model.train()
	optimizer.zero_grad()        
	_, batch1 = trainloader1_iter.__next__()
	_, batch2 = trainloader2_iter.__next__()
	images1, labels1 = batch1
	images2, labels2 = batch2
	loss1 = criterion1(outputs1, labels1) 
	loss2 = criterion2(outputs2, labels2)
	loss = loss1 + loss2
	loss.backward()
	optimizer.step()
2

In the default setup your loss will be the average or sum of your batch.
You can set reduce=False to get the same batch size for your loss:

batch_size = 10
criterion = nn.MSELoss(reduce=False)
x = torch.randn(batch_size, 2, requires_grad=True)
y = torch.randn(batch_size, 2)
loss = criterion(x, y)
3

Thank you. So my loss is for batch size of 16, Am I right? Because I want the network compute grad. for batch size of 16. And loss is loss1 + loss2 so I cannot set up reduction flag in the loss

4

I’m not sure I’m understanding your issue correctly.
You could just use the standard loss function using the average of your batch loss, sum the losses together, and call backward on the sum of losses.

5

Yes. I have two losses. Each loss take 16 images as a batch. Then the total loss is sum of the two losses. The backward is called for the total loss. So, I am not sure the total loss is performing on batch size of 16 or 32 ( two batch from two losses). Are you clear my problem?