I mean something like a key value pair between your color codes in RGB and the corresponding class index.
E.g. [128, 64, 128] would map to class0.
Have a look at this post for another example using grayscale images.
Could you post the classes for each separate color in your segmentation mask?
2 Likes