I now have PyToch 1.6 / TorchVision 0.7 for 64 bit Raspberry OS posted on my page linked above. Compiling PyTorch on the 8GB Raspberry Pi worked out of the box (no swap needed, no memory problems) and took a bit less than 4 hours (the .git checkout and the whl have timestamps exactly 4 hours apart, but I did the installation of requirements in between, too).
I hope you enjoy it.
Best regards
Thomas