Docker ccache rebuild bug

I am currently building PyTorch from source using a Dockerfile with BuildKit enabled.
I am using CCache to cache my builds.
Although CCache does increase the build speed, I find that the project is rebuilt even when the same branch is checked out. Is this to be expected?
I include a link to my project. This issue does not occur in the same run but between different Docker builds.