Rescaling Labels in Semantic Segmentation

In order to apply Random Scaling and Cropping as a data preprocessing step in Semantic Segmentation, what interpolation should we use for labels?

I am currently using Image.NEAREST from PIL but my labels get messed up after interpolation. (only displaying the labels for plane). This is an Image from PASCALVOC dataset.

Snip20171116_1Snip20171116_2

My Bad! Even the original ground truth label is distorted.Snip20171116_3