Pip’s tendency to (re)install dependencies already installed tends to be greater than my own. If you feel similar, you could use --no-deps when installing torchvision.
Installing torchvision from github may also be a good alternative, it is far easier than for torch. See https://github.com/pytorch/vision .
Best regards
Thomas