You might be running into the misleading output of Windows’ task manager as explained here and here. Use nvidia-smi or select the right task manager view.
nvidia-smi