Pytorch segmentation faults when eGPU is not available

I have an external GPU connected to my computer, and I have GPU version of pytorch installed, and it works fine. But whenever the GPU is not connected, as soon as I import pytorch, it crashes with the message “segmentation fault (core dumped)”. So even if I want to run pytorch on just CPU, I can not do it if the eGPU is not connected.

OS: Ubuntu 16.04.4 LTS
eGPU: TITAN X (Pascal)
Nvidia Driver Version: 390.48
Cuda compilation tools, release 9.1, V9.1.85
torch.version: 0.4.0