I think that you are initializing the network before the dataloader. For this reason, when you change the network size, the samples generated by the dataloader also change. Because, as you know all filters and bias need to be initialized normally using random methods. A change of the number of times that random method is used implies a change to the state of the random generator and this can be the cause that your dataloader is generating different values.
Great example of never tag.
As @amosella explains much better than I would have, the random init leaves you with different states depending on the number of parameters (and possibly the init method).
So the easiest solution is to re-seed after the weight init and before the dataloader init. If I remember correctly, the CPU random generator is the important one here.
In my case, I had two versions of the code loading same data (differently stored). I expected the network to be the same but still the data was shuffled. However, the above hack worked if I re-seeded before the dataloader code. Not sure what was causing the issue but it worked.