It is definitely possible to build from source:
If you are not using MAGMA it is also possible to remove it from the build e.g., via the USE_MAGMA=0 environment variable when running the build command.
Alternatively, if you are OK with using a somewhat older version to avoid building from source, it looks like sm_35 should be supported by default in the published binaries to at least 1.8.1+, according to: GPU compute capability support for each pytorch version - #7 by moi90