Different input for the same network for pytorch

#1

Hello.
Three different input: images1, images1_sub(a subset of images1), images2,.
One network: model.

... ...
optimizer.zero_grad()

output1，intermediate_feature_map_1  = model(images1)
# get output and intermediate feature map
output1_sub, intermediate_feature_map_sub1 = model(images1_sub)
# get output and intermediate feature map
output2, intermediate_feature_map2 = model(images2)

# calculate the cross_entropy ce_loss1 for output1
ce_loss1 = cross_entropy(output1, label)
# calculate the cross_entropy ce_loss2 for output2
ce_loss2 = cross_entropy(output2, label)  # the same label
# the distance of intermediate feature map for images_sub1 and images2 
#(the shape of images_sub1 is the same as the shape of images2  )
loss_d =  distance(intermediate_feature_map_sub1, intermediate_feature_map2 )
loss =  ce_loss1 + ce_loss2 + loss_d 
loss.backward()
optimizer.step()

I’d like to sum the loss and backward. Then update parameter for network.
Is there anything wrong with my code?

Appreciated for your help!

#2

The code looks alright.
Do you see any strange things happening?

1 Like
#4

Hi, thanks for your reply!
I have not seen anything strange happening so far. But I have a question:
images1_sub in the code above is a subset of images1. If images1_sub = images1[:half]
Compared to the code:
output1_sub, intermediate_feature_map_sub1 = model(images1_sub)
will the following code achieve the same goal and save some calculation resouces?
output1_sub, intermediate_feature_map_sub1 = output1[:half], intermediate_feature_map1[:half]
Thank you.

#5

This won’t necessarily give the same output.
E.g. if you are using conv layers with some padding, the outputs might differ:

conv = nn.Conv2d(1, 1, 3, 1, 1)
x = torch.randn(1, 1, 6, 6)
output = conv(x)
output_half = conv(x[:, :, 3:, 3:])

print(output[:, :, 3:, 3:] == output_half)
> tensor([[[[0, 0, 0],
            [0, 1, 1],
            [0, 1, 1]]]], dtype=torch.uint8)
1 Like
#7

I got it. Thanks for your reply!

#8

Yeah, :grinning: but, I think my original question is : about half batch of images ~

images1_sub = images1[:half]
output1_sub, intermediate_feature_map_sub1 = model(images1_sub)

and

outptut, inter = model(images)
output_half, inter_half = outptut[:half], inter[:half]

Is output1_sub different from output_half ? I have tested it, maybe same?

#9

Thanks for the information. I though you would like to split the spatial size.
If you split the batch, you would have to take care of e.g. batch norm layers, as the running stats will differ.
Besides that it should yield the same output.

What results did your test yield?

#10

Yes, with batch norm layers, I tested dummy data. The output is different. Appreciate for your reply!