Wrong nvcc version when compiling plugins

I do have a 2.3.1 installation on a conda env in windows with cuda 12.6. I do also have a systemwide 12.6 and 11.x installation. I tried to run stylegan 3 on this environment but getting errors when compiling plugins. It tries to use cuda11.x and throws an error with version mismatch for vstudio 2022 latest update. I just want it to use cuda 12.6 on environment level. On the logs, it goes for systemwide cuda 11.x installation and when I change the filepath of cuda11.x system wide, throws path error not switching to other paths. To be precise:

RuntimeError: Error building extension ‘bias_act_plugin’: [1/2] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\bin\nvcc --generate-dependencies-with-compile --dependency-output bias_act.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\Users\clt\miniforge3\envs\pytorch_2.3.1\Lib\site-packages\torch\include -IC:\Users\clt\miniforge3\envs\pytorch_2.3.1\Lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\clt\miniforge3\envs\pytorch_2.3.1\Lib\site-packages\torch\include\TH -IC:\Users\clt\miniforge3\envs\pytorch_2.3.1\Lib\site-packages\torch\include\THC “-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\include” -IC:\Users\clt\miniforge3\envs\pytorch_2.3.1\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 --use_fast_math --allow-unsupported-compiler -c C:\Users\clt\AppData\Local\torch_extensions\torch_extensions\Cache\py312_cu121\bias_act_plugin\d197188c6eb43b2685156af41c1c4adc-nvidia-geforce-rtx-3060\bias_act.cu -o bias_act.cuda.o
FAILED: bias_act.cuda.o