Differing output for model.eval() vs model.train()

1

I’m currently implementing a DETR head for my model, to replace a semantic segmentation head. However, I’m getting strange results for the bboxes when in model.eval() where they’re all clustered closely together around the middle of the image, whereas in model.train() they seem reasonably spread out and actually detecting things.

Gradients on/off has no effect. There are no branches in the model code for module.training or anything like that, so I’m unsure why this occuring and where to look to resolve this; I’ve never had this issue before…

Ubuntu20.04 + Pytorch 1.7.1 + RTX3090 (CUDA 11.1)

2

If possible I’d like to see your code

3

Github repo and branch (feature/detr_head) is linked below, the model definition can be found in nnet_training/nnet_models/detr_sfd.py

4

Sorry for the late reply.
There’s nothing here tho

5

Yeah sorry I merged the branch into master. I’m using deeplab’s panotpic architecture rather than DETR, which worked fine straight away, no boundary box schenanigans. And since that’s working now I’m coming up with my own arch which will better unify panoptic segmentation into one singlular pipeline (No results yet, still working on it).

Link to the master branch is below.