Data augmentation in semantic segmentation

vision
1

Hi again
I just want to ask about these lines in semantic segmentation data augmentation operation
based on previews question discussion ptrblck said :

Spatial transformations applied on the input image should also be applied on the mask tensor to make sure that the input pixel location still corresponds to the mask (e.g. rotations, resizing, etc.).
Besides that, you should treat the mask as a classification target and should not change its values.

also, I found this statement in Albumentations data augmentation methods:
Many non-spatial transformations like,CLAHE, [RandomBrightness], [RandomContrast] [RandomGamma] can be also added. They will be applied only to the image and not the mask.

and they writ this code :

aug = A.Compose([
A.OneOf([
A.RandomSizedCrop(min_max_height=(50, 101), height=original_height, width=original_width, p=0.5),
A.PadIfNeeded(min_height=original_height, min_width=original_width, p=0.5)
], p=1),
A.VerticalFlip(p=0.5),
A.RandomRotate90(p=0.5),
A.OneOf([
A.ElasticTransform(alpha=120, sigma=120 * 0.05, alpha_affine=120 * 0.03, p=0.5),
A.GridDistortion(p=0.5),
A.OpticalDistortion(distort_limit=2, shift_limit=0.5, p=1)
], p=0.8),
A.CLAHE(p=0.8),
A.RandomBrightnessContrast(p=0.8),
A.RandomGamma(p=0.8)])

random.seed(11)
augmented = aug(image=image, mask=mask)

image_heavy = augmented[‘image’]
mask_heavy = augmented[‘mask’]

visualize(image_heavy, mask_heavy, original_image=image, original_mask=mask)

my question is: based on the above code the augment operation applied to the image and mask why they write “they will be applied only to the image and not the mask”

2

Yes, because some image transformations which involve change of the intensity values of the original image, such as contrast enhancement, randomGamma etc. need to be only applied to the image.

Applying intensity based transformations to the mask does not make sense since the mask is composed of discrete values 0,1,2 etc. and it makes no sense to apply intensity values to mask as it will destroy the mask values.

Wheres, spatial transformations such as flipping the image, or rotating the image should be applied to the mask as well, since the orientation of the image is changed which changes the orientation of the corresponding mask also.

Is this clear?

1 Like
3

Yep, It is clear , Thank you
but in code they apply the transformation to both image and mask , Why ?
this my question?

4

Which line exactly in your original post. Can you tell?

5

mask_heavy = augmented[‘mask’]

6

Here they call the augment and the mask value changed

7

I think the heavy() transformation may be a spatial augmentation (such as rotate(), flip() etc.) that is why the mask and image both are being changed.

8

but here we change the contrast and adding some noise to mask

9

I think there is some misunderstanding. Noise is never added to the mask, noise is only added to the image. Since, think about it, if you add noise to the mask, the mask values get changed and the mask will stop making sense. Did you understand?

1 Like
10

I understand dear Megh_Bhalerao
but I am asking about these statements A.CLAHE(p=0.8),
A.RandomBrightnessContrast(p=0.8),
A.RandomGamma(p=0.8), are these statements change the mask value or not? because changing contrast seems to be adding some value to the mask.

11

No, they will not modify the mask in any way as far as my understanding goes. Since Brightness and Gamma are intensity based transformations. I am not sure why you are observing the a change in the mask values. Maybe you can double check your code.

12

ok, because of this statement I am confused :
“Many non-spatial transformations like,CLAHE, [RandomBrightness], [RandomContrast] [RandomGamma] can be also added. They will be applied only to the image and not the mask.”
they said will be only applied to the image, not the mask but they applied to both using this :
image_heavy = augmented[‘image’]
mask_heavy = augmented[‘mask’]
this is my mistake

13

image_heavy = augmented[‘image’]
mask_heavy = augmented[‘mask’]

This is just representative. Some aug will be added to mask and some will not.

14

How can I know which one added to image and which one will added to mask if they call augment
am I just call augment or i have to specify which one will be called.

15

No, you do not need to specify. It will automatically know.

1 Like
16

this means we can add any type of augmentation in the function above and during these two lines:
image_heavy = augmented[‘image’]
mask_heavy = augmented[‘mask’]
some operations will apply to images and others to mask based on the type of augmentation.
is this right?

17

Yes. That is correct.

1 Like
18

ok
thank you so much
I appreciate that