How to stabilize a simple network to small changes in input?

Beginner question here: I have a small simple fully connected network (1-3 layers, 50 hidden nodes) that is training properly, but I’m getting different results with small changes in input data. The network is deterministic (I have seeded everythig properly, different runs in same data yield exact same result) so that’s not it.

How can I train my simple network to be more robust? I have tried the simple obvious things like playing around with batch size and number of hidden nodes. What else can I look into? Thanks.

I would guess the sensitivity to small changes in the input data depends on the signal and noise in the input data.
E.g. if your input signal has a very low magnitude, your model would have to learn these small differences to correctly predict the output.
On the other hand, you might try to add noise to the input to make the model more robust.