There is a similar issue here. A lot of stuff might be happening.
- Are you sure your labels are correct?
- Did you try other learning rates? smaller? bigger?
- Did you try other architectures? You might be suffering from under-fitting.
- Did you try other losses?
- Did you normalize your training data?