RuntimeError: CUDA Memory error

I am training on different image sizes as compared to the validation data set image sizes. When the image sizes in both train and valid are same, the lightning module framework runs without errors, otherwise it throws the below error:

RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Can someone advise the solution?
Thanks!