Pytorch docker image significant size increase in newer version


I am building a Docker image for training a mask2former and notice a significant increase in base image size:
pytorch/pytorch 2.1.1-cuda12.1-cudnn8-devel 0b705662863d 2 months ago 16.6GB
pytorch/pytorch 2.0.1-cuda11.7-cudnn8-devel 42a0e9b621e2 8 months ago 13.2GB

In the end, adding my dependencies (transformers, lightning and a few others), and keeping everything else equal I get a final image ~15GB for the older base image and ~ 20GB for the latest. I also notice that other allude to the issue

I am unsure how I should approach the topic if I want to understand the reason for the different. Does anyone have insights that can help me on the way?

Of course I would like to be on the latest version but 20GB prohibitively larger for my training deployment