I am seeing exactly the same thing! I am doing a detection network, and the determinism is only happening in the first step. From second step it starts to diverge from model to model, and in the end it results in very different final models.
I am also using interpolation function.