Are you transforming the target image with the provided transformation or just the data tensor?
In the former case, you will get a normalized target, which is wrong for a segmentation use case.
You would have to map the colors of your target to class indices.
E.g. “blue” -> class0, “red” -> class1 etc.
Here is an example on how to create a color mapping.