Rescaling Labels in Semantic Segmentation

In order to apply Random Scaling and Cropping as a data preprocessing step in Semantic Segmentation, what interpolation should we use for labels?

I am currently using Image.NEAREST from PIL but my labels get messed up after interpolation. (only displaying the labels for plane). This is an Image from PASCALVOC dataset.


My Bad! Even the original ground truth label is distorted.Snip20171116_3