When epochs loss causing on train/test score

In NN training when epochs loss decrease monotonically ,then score on train and validation data must improve. Its not happening consistently , some time score improve when epochs loss decrease monotonically sometime it does not decrease and gets worst. I am using binary cross entropy , How can i rely on loss rather than train/test score?