This was actually an error with Visual Stufio being installed incorrectly. I uninstalled all Vsiual Studio components, removed the environtment variables (named ones and from Path), and removed all files on my disk. Then I installed VS 2019 (As that is the latest supported version by PyTorch as of making this post) and installed the C++ dev tools (SDK, build tools) with it. I then had to re-install Pytorch in conda (conda uninstall pytorch torchvision torchaudio cudatoolkit) and then installed them again, simply replacing the existing packages did not work.
After all this, it finally worked.